httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jim Jagielski <...@jaguNET.com>
Subject Re: collaboration request - apache server automation testing with swat tool
Date Sat, 06 Feb 2016 14:34:58 GMT

> On Feb 5, 2016, at 9:26 AM, Alexey Melezhik <melezhik@gmail.com> wrote:
> 
>> I wonder how possible it would be to transcode the old tests to Swat?
> 
> Swat has some architectural specifics. As it simply makes http
> requests ( by curl ) and validate server response ( by swat DSL ) it
> is considered as black box testing tools , there is no way to interact
> with httpd internal structure. So probably most of old tests could be
> easily ported but some probably not ( which implies something more
> complicated ).
> 

Most of the other tests w/ the old framework is the exact same...
requests sent and responses verified against what is expected.

> Basically  swat could cover simple and average use cases, everything
> that could  expressed in terms of make http requests and then
> analyzing an output , see examples for some issues I have expressed in
> swat scenarios at https://github.com/melezhik/apache-swat
> 
> 
> Also alternatively we may run 2 frameworks in parallel. Apache-swat is
> intended for developers usage first and is able to run per-issues
> test.
> 
> This how I see possible  workflow:
> 
> * developer accept a bug request
> * developer ( or some else ) create swat test for given bug/feature
> * so we have `test first` approach - everyone involved in bug/feature
> resolution has possibility to verify changes by simply running :
> 
> swat -t $issue-id
> 
> 
> 
> 
> 
> 
> 2016-02-05 17:08 GMT+03:00 Alexey Melezhik <melezhik@gmail.com>:
>> 2016-02-05 17:01 GMT+03:00 Jim Jagielski <jim@jagunet.com>:
>>> Personally, I like the idea of having another framework;
>>> the current one is OK but somewhat "painful" to update.
>> 
>> Hi! What kind of possible issues you here ? when saying "painful" to
>> update, please explain.
>> Thanks.
>> 
>> Also , please consider this link - https://github.com/melezhik/apache-swat and
>> this post - http://mail-archives.apache.org/mod_mbox/httpd-dev/201602.mbox/%3CCAL7UUk-zqb1=7wDvCe8E-Ku=U=aY0MdJ_-JgCjVzZdvAranKQg@mail.gmail.com%3E
>> as latest version of this idea ....
>> 
>> 
>> 
>> 
>> 
>>> 
>>> I wonder how possible it would be to transcode the old tests
>>> to Swat? We could then provide for 2 testing frameworks, one
>>> developed by the ASF and the other external and 3rd party.
>>> 
>>> 
>>>> On Feb 1, 2016, at 5:23 AM, Alexey Melezhik <melezhik@gmail.com> wrote:
>>>> 
>>>> Here the list of existed issues I was able to automate tests for:
>>>> 
>>>> vagrant@Debian-jessie-amd64-netboot:~/my/apache-swat$ ls -1 | grep -P '\d'
>>>> 44221
>>>> 46751
>>>> 58789
>>>> 58828
>>>> 58854
>>>> 
>>>> 
>>>> I have informed developers at https://bz.apache.org/bugzilla/ about
>>>> these, so they rely upon them. Or you have some pre-release testing,
>>>> so run all of these in a whole chunk?
>>>> 
>>>> Regards
>>>> 
>>>> Alexey
>>>> 
>>>> PS chaining issues / tests listing  as always -
>>>> https://github.com/melezhik/apache-swat
>>>> 
>>>> PS2 it'd be could to verify tests as well against "bleeding edge"
>>>> apache version gets installed from SCM, but I don't know how to do
>>>> this.
>>>> Currently I use http://httpd.apache.org/download.cgi ( last stable release
)
>>>> 
>>>> 
>>>> 
>>>> 2016-01-30 21:36 GMT+03:00 Alexey Melezhik <melezhik@gmail.com>:
>>>>> Hi Bill!
>>>>> 
>>>>> I have started to assemble swat tests  for apache2 issues coming from
>>>>> https://bz.apache.org/bugzilla
>>>>> 
>>>>> And this the first one - https://bz.apache.org/bugzilla/show_bug.cgi?id=44221
,
>>>>> 
>>>>> Test suite is failed for the moment :
>>>>> 
>>>>> 
>>>>> vagrant@Debian-jessie-amd64-netboot:~/my/apache-swat$ swat -t 44221
>>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.baz/00.GET.t ...
>>>>> ok 1 - GET 127.0.0.1/44221/foo.baz succeeded
>>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/52jLAo76YZ.hdr
>>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/52jLAo76YZ
>>>>> ok 2 - output match '200 OK'
>>>>> 1..2
>>>>> ok
>>>>> /home/vagrant/.swat/.cache/384/prove/44221/FOO.bar/00.GET.t ...
>>>>> ok 1 - GET 127.0.0.1/44221/FOO.bar succeeded
>>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/DvEZnO6ALe.hdr
>>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/DvEZnO6ALe
>>>>> ok 2 - output match '200 OK'
>>>>> ok 3 - output match /Location: \S+/
>>>>> ok 4 - 'Location: http://127.0.0.1/44221/foo.bar' match 'foo.bar'
>>>>> 1..4
>>>>> ok
>>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.BAR/00.GET.t ...
>>>>> ok 1 - GET 127.0.0.1/44221/foo.BAR succeeded
>>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/bWQ1Sbl0YZ.hdr
>>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/bWQ1Sbl0YZ
>>>>> ok 2 - output match '200 OK'
>>>>> ok 3 - output match /Location: \S+/
>>>>> ok 4 - 'Location: http://127.0.0.1/44221/foo.bar' match 'foo.bar'
>>>>> 1..4
>>>>> ok
>>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.bar/00.GET.t ...
>>>>> ok 1 - GET 127.0.0.1/44221/foo.bar succeeded
>>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/Jd_hlo7Qwv.hdr
>>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/Jd_hlo7Qwv
>>>>> ok 2 - output match '200 OK'
>>>>> 1..2
>>>>> ok
>>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.html/00.GET.t ..
>>>>> ok 1 - GET 127.0.0.1/44221/foo.html succeeded
>>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/tW2L511eym.hdr
>>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/tW2L511eym
>>>>> ok 2 - output match /HTTP\/(\S+) (\d+) \S+/
>>>>> not ok 3 - 'HTTP/1.1 300 Multiple Choices' match /404 /
>>>>> 
>>>>> #   Failed test ''HTTP/1.1 300 Multiple Choices' match /404 /'
>>>>> #   at /usr/local/share/perl/5.20.2/swat.pm line 218.
>>>>> not ok 4 - output match 'Not Found'
>>>>> 
>>>>> #   Failed test 'output match 'Not Found''
>>>>> #   at /usr/local/share/perl/5.20.2/swat.pm line 218.
>>>>> 1..4
>>>>> # Looks like you failed 2 tests of 4.
>>>>> Dubious, test returned 2 (wstat 512, 0x200)
>>>>> Failed 2/4 subtests
>>>>> 
>>>>> Test Summary Report
>>>>> -------------------
>>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.html/00.GET.t (Wstat:
>>>>> 512 Tests: 4 Failed: 2)
>>>>> Failed tests:  3-4
>>>>> Non-zero exit status: 2
>>>>> Files=5, Tests=16,  0 wallclock secs ( 0.03 usr  0.00 sys +  0.27 cusr
>>>>> 0.00 csys =  0.30 CPU)
>>>>> Result: FAIL
>>>>> 
>>>>> ...
>>>>> 
>>>>> Ok, this is just the beginning ;-))) , watch soon updates at -
>>>>> https://github.com/melezhik/apache-swat
>>>>> 
>>>>> 2016-01-29 20:44 GMT+03:00 William A Rowe Jr <wrowe@rowe-clan.net>:
>>>>>> On Fri, Jan 29, 2016 at 6:35 AM, Alexey Melezhik <melezhik@gmail.com>
wrote:
>>>>>>> 
>>>>>>> Hi Bill!
>>>>>>> 
>>>>>>> Any news? ( Please see my previous reply ...)
>>>>>>> Intr
>>>>>> 
>>>>>> 
>>>>>> Intrigued :)  But my responses will be delayed, I personally
>>>>>> won't have time to look further myself until other backlogged
>>>>>> commitments to httpd are caught up some more.
>>>>>> 
>>>>>> Be aware many of us have only a few hours to participate in the
>>>>>> httpd project itself, so it will take several days or more to gather
>>>>>> feedback on your interest and perhaps a champion to drive such
>>>>>> an effort if it is actively driven from this side.  Of course that
>>>>>> should not stop you from assembling some ideas at the vcs of
>>>>>> your choice, and sharing a link for those who want to look at an
>>>>>> early draft of such an effort!
>>>>>> 
>>>>>> Cheers,
>>>>>> 
>>>>>> Bill
>>> 


Mime
View raw message