httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexey Melezhik <melez...@gmail.com>
Subject Re: collaboration request - apache server automation testing with swat tool
Date Fri, 05 Feb 2016 14:26:39 GMT
> I wonder how possible it would be to transcode the old tests to Swat?

Swat has some architectural specifics. As it simply makes http
requests ( by curl ) and validate server response ( by swat DSL ) it
is considered as black box testing tools , there is no way to interact
with httpd internal structure. So probably most of old tests could be
easily ported but some probably not ( which implies something more
complicated ).

Basically  swat could cover simple and average use cases, everything
that could  expressed in terms of make http requests and then
analyzing an output , see examples for some issues I have expressed in
swat scenarios at https://github.com/melezhik/apache-swat


Also alternatively we may run 2 frameworks in parallel. Apache-swat is
intended for developers usage first and is able to run per-issues
test.

This how I see possible  workflow:

* developer accept a bug request
* developer ( or some else ) create swat test for given bug/feature
* so we have `test first` approach - everyone involved in bug/feature
resolution has possibility to verify changes by simply running :

 swat -t $issue-id






2016-02-05 17:08 GMT+03:00 Alexey Melezhik <melezhik@gmail.com>:
> 2016-02-05 17:01 GMT+03:00 Jim Jagielski <jim@jagunet.com>:
>> Personally, I like the idea of having another framework;
>> the current one is OK but somewhat "painful" to update.
>
> Hi! What kind of possible issues you here ? when saying "painful" to
> update, please explain.
> Thanks.
>
> Also , please consider this link - https://github.com/melezhik/apache-swat and
> this post - http://mail-archives.apache.org/mod_mbox/httpd-dev/201602.mbox/%3CCAL7UUk-zqb1=7wDvCe8E-Ku=U=aY0MdJ_-JgCjVzZdvAranKQg@mail.gmail.com%3E
> as latest version of this idea ....
>
>
>
>
>
>>
>> I wonder how possible it would be to transcode the old tests
>> to Swat? We could then provide for 2 testing frameworks, one
>> developed by the ASF and the other external and 3rd party.
>>
>>
>>> On Feb 1, 2016, at 5:23 AM, Alexey Melezhik <melezhik@gmail.com> wrote:
>>>
>>> Here the list of existed issues I was able to automate tests for:
>>>
>>> vagrant@Debian-jessie-amd64-netboot:~/my/apache-swat$ ls -1 | grep -P '\d'
>>> 44221
>>> 46751
>>> 58789
>>> 58828
>>> 58854
>>>
>>>
>>> I have informed developers at https://bz.apache.org/bugzilla/ about
>>> these, so they rely upon them. Or you have some pre-release testing,
>>> so run all of these in a whole chunk?
>>>
>>> Regards
>>>
>>> Alexey
>>>
>>> PS chaining issues / tests listing  as always -
>>> https://github.com/melezhik/apache-swat
>>>
>>> PS2 it'd be could to verify tests as well against "bleeding edge"
>>> apache version gets installed from SCM, but I don't know how to do
>>> this.
>>> Currently I use http://httpd.apache.org/download.cgi ( last stable release )
>>>
>>>
>>>
>>> 2016-01-30 21:36 GMT+03:00 Alexey Melezhik <melezhik@gmail.com>:
>>>> Hi Bill!
>>>>
>>>> I have started to assemble swat tests  for apache2 issues coming from
>>>> https://bz.apache.org/bugzilla
>>>>
>>>> And this the first one - https://bz.apache.org/bugzilla/show_bug.cgi?id=44221
,
>>>>
>>>> Test suite is failed for the moment :
>>>>
>>>>
>>>> vagrant@Debian-jessie-amd64-netboot:~/my/apache-swat$ swat -t 44221
>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.baz/00.GET.t ...
>>>> ok 1 - GET 127.0.0.1/44221/foo.baz succeeded
>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/52jLAo76YZ.hdr
>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/52jLAo76YZ
>>>> ok 2 - output match '200 OK'
>>>> 1..2
>>>> ok
>>>> /home/vagrant/.swat/.cache/384/prove/44221/FOO.bar/00.GET.t ...
>>>> ok 1 - GET 127.0.0.1/44221/FOO.bar succeeded
>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/DvEZnO6ALe.hdr
>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/DvEZnO6ALe
>>>> ok 2 - output match '200 OK'
>>>> ok 3 - output match /Location: \S+/
>>>> ok 4 - 'Location: http://127.0.0.1/44221/foo.bar' match 'foo.bar'
>>>> 1..4
>>>> ok
>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.BAR/00.GET.t ...
>>>> ok 1 - GET 127.0.0.1/44221/foo.BAR succeeded
>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/bWQ1Sbl0YZ.hdr
>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/bWQ1Sbl0YZ
>>>> ok 2 - output match '200 OK'
>>>> ok 3 - output match /Location: \S+/
>>>> ok 4 - 'Location: http://127.0.0.1/44221/foo.bar' match 'foo.bar'
>>>> 1..4
>>>> ok
>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.bar/00.GET.t ...
>>>> ok 1 - GET 127.0.0.1/44221/foo.bar succeeded
>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/Jd_hlo7Qwv.hdr
>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/Jd_hlo7Qwv
>>>> ok 2 - output match '200 OK'
>>>> 1..2
>>>> ok
>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.html/00.GET.t ..
>>>> ok 1 - GET 127.0.0.1/44221/foo.html succeeded
>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/tW2L511eym.hdr
>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/tW2L511eym
>>>> ok 2 - output match /HTTP\/(\S+) (\d+) \S+/
>>>> not ok 3 - 'HTTP/1.1 300 Multiple Choices' match /404 /
>>>>
>>>> #   Failed test ''HTTP/1.1 300 Multiple Choices' match /404 /'
>>>> #   at /usr/local/share/perl/5.20.2/swat.pm line 218.
>>>> not ok 4 - output match 'Not Found'
>>>>
>>>> #   Failed test 'output match 'Not Found''
>>>> #   at /usr/local/share/perl/5.20.2/swat.pm line 218.
>>>> 1..4
>>>> # Looks like you failed 2 tests of 4.
>>>> Dubious, test returned 2 (wstat 512, 0x200)
>>>> Failed 2/4 subtests
>>>>
>>>> Test Summary Report
>>>> -------------------
>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.html/00.GET.t (Wstat:
>>>> 512 Tests: 4 Failed: 2)
>>>>  Failed tests:  3-4
>>>>  Non-zero exit status: 2
>>>> Files=5, Tests=16,  0 wallclock secs ( 0.03 usr  0.00 sys +  0.27 cusr
>>>> 0.00 csys =  0.30 CPU)
>>>> Result: FAIL
>>>>
>>>> ...
>>>>
>>>> Ok, this is just the beginning ;-))) , watch soon updates at -
>>>> https://github.com/melezhik/apache-swat
>>>>
>>>> 2016-01-29 20:44 GMT+03:00 William A Rowe Jr <wrowe@rowe-clan.net>:
>>>>> On Fri, Jan 29, 2016 at 6:35 AM, Alexey Melezhik <melezhik@gmail.com>
wrote:
>>>>>>
>>>>>> Hi Bill!
>>>>>>
>>>>>> Any news? ( Please see my previous reply ...)
>>>>>> Intr
>>>>>
>>>>>
>>>>> Intrigued :)  But my responses will be delayed, I personally
>>>>> won't have time to look further myself until other backlogged
>>>>> commitments to httpd are caught up some more.
>>>>>
>>>>> Be aware many of us have only a few hours to participate in the
>>>>> httpd project itself, so it will take several days or more to gather
>>>>> feedback on your interest and perhaps a champion to drive such
>>>>> an effort if it is actively driven from this side.  Of course that
>>>>> should not stop you from assembling some ideas at the vcs of
>>>>> your choice, and sharing a link for those who want to look at an
>>>>> early draft of such an effort!
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Bill
>>

Mime
View raw message