httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexey Melezhik <melez...@gmail.com>
Subject Re: collaboration request - apache server automation testing with swat tool
Date Sat, 06 Feb 2016 16:03:24 GMT
Sure, then it's probably won't be big deal to port existed test code base.

Here I want to highlight some swat features which make it more then
just engine to make http requests and verify an output.

Swat adds some features  could be of interest in httpd2 testing, to
describe briefly  a few:

- *Curl centric design *

Swat relies on curl heavily, constructing all http requests in a terms
of curl command line utility and then passing them to curl to make
http request. This a  single swat test unit is based  on more or many
http request made by curl. IMHO using curl has some essential benefits
for httpd2 testing, to list a few:

     - often apache server testing implies low level http context
which is out of the box when using cur. Consider a few cases:
        - making http2 protocol requests ( curl does! )
        - setting explicit timeouts ( per initial connection and per
whole requests )
        - issuing "raw" http requests with chunked/broken data to
tests how apache reacts on invalid requests
        - construction of various html related entities  ( forms ,
cookies , named parameters )
        - various types of authentication works out of the box -
--basic, --digest, --ntlm, and --negotiate
        - ssl related testing ( not relates to http but ... )

Also please take into account:

- curl is stable and well supported
- no need to install to many perl dependencies for http related stuff
- all the features granted by latest bleeding edge versions of curl
are here to use

- *Simple debugging*.

As swat uses curl to make http request. _basically_ a failing test
mean you may reproduce the same case manually using curl as you have
all you need here ( http request parameters get dumped into console )
In other words swat is much closer to what people usually do when
carry out manual  testing - get a grip with curl, send command and
then see the output. Such an approach makes it swat's usage extremely
simple and understandable.


- *Sequential curl requests*.

As curl a request oriented ( only one single request could be made at
time ) it has some limits. As swat just uses curl it makes it possible
to create so called sequential/compound requests. When before one
request other request is issued. As trivial example of such a case is
user authentication. When one needs access for restricted http
resources and a preliminary authentication is required.  So first we
need get logged in and only then request a restricted resource. Such a
compound requests on other hand are just a single test story for end
user and are hidden under the hood wheh using swat - see
https://github.com/melezhik/swat#upstream-and-downstream-stories for
details.

- * Text validation DSL *

And finally swat provides  rich and simple DSL for text validation.
This simplifies server output verification and _often_ even does not
require a single line of perl code ( as it none perlish DSL ).  In
other hand swat is extendable by perl to add desired complexity to
your tests. Take a look at examples - https://github.com/melezhik/swat
and https://github.com/melezhik/swat/tree/master/examples

Regards

Alexey


2016-02-06 17:34 GMT+03:00 Jim Jagielski <jim@jagunet.com>:
>
>> On Feb 5, 2016, at 9:26 AM, Alexey Melezhik <melezhik@gmail.com> wrote:
>>
>>> I wonder how possible it would be to transcode the old tests to Swat?
>>
>> Swat has some architectural specifics. As it simply makes http
>> requests ( by curl ) and validate server response ( by swat DSL ) it
>> is considered as black box testing tools , there is no way to interact
>> with httpd internal structure. So probably most of old tests could be
>> easily ported but some probably not ( which implies something more
>> complicated ).
>>
>
> Most of the other tests w/ the old framework is the exact same...
> requests sent and responses verified against what is expected.
>
>> Basically  swat could cover simple and average use cases, everything
>> that could  expressed in terms of make http requests and then
>> analyzing an output , see examples for some issues I have expressed in
>> swat scenarios at https://github.com/melezhik/apache-swat
>>
>>
>> Also alternatively we may run 2 frameworks in parallel. Apache-swat is
>> intended for developers usage first and is able to run per-issues
>> test.
>>
>> This how I see possible  workflow:
>>
>> * developer accept a bug request
>> * developer ( or some else ) create swat test for given bug/feature
>> * so we have `test first` approach - everyone involved in bug/feature
>> resolution has possibility to verify changes by simply running :
>>
>> swat -t $issue-id
>>
>>
>>
>>
>>
>>
>> 2016-02-05 17:08 GMT+03:00 Alexey Melezhik <melezhik@gmail.com>:
>>> 2016-02-05 17:01 GMT+03:00 Jim Jagielski <jim@jagunet.com>:
>>>> Personally, I like the idea of having another framework;
>>>> the current one is OK but somewhat "painful" to update.
>>>
>>> Hi! What kind of possible issues you here ? when saying "painful" to
>>> update, please explain.
>>> Thanks.
>>>
>>> Also , please consider this link - https://github.com/melezhik/apache-swat and
>>> this post - http://mail-archives.apache.org/mod_mbox/httpd-dev/201602.mbox/%3CCAL7UUk-zqb1=7wDvCe8E-Ku=U=aY0MdJ_-JgCjVzZdvAranKQg@mail.gmail.com%3E
>>> as latest version of this idea ....
>>>
>>>
>>>
>>>
>>>
>>>>
>>>> I wonder how possible it would be to transcode the old tests
>>>> to Swat? We could then provide for 2 testing frameworks, one
>>>> developed by the ASF and the other external and 3rd party.
>>>>
>>>>
>>>>> On Feb 1, 2016, at 5:23 AM, Alexey Melezhik <melezhik@gmail.com>
wrote:
>>>>>
>>>>> Here the list of existed issues I was able to automate tests for:
>>>>>
>>>>> vagrant@Debian-jessie-amd64-netboot:~/my/apache-swat$ ls -1 | grep -P
'\d'
>>>>> 44221
>>>>> 46751
>>>>> 58789
>>>>> 58828
>>>>> 58854
>>>>>
>>>>>
>>>>> I have informed developers at https://bz.apache.org/bugzilla/ about
>>>>> these, so they rely upon them. Or you have some pre-release testing,
>>>>> so run all of these in a whole chunk?
>>>>>
>>>>> Regards
>>>>>
>>>>> Alexey
>>>>>
>>>>> PS chaining issues / tests listing  as always -
>>>>> https://github.com/melezhik/apache-swat
>>>>>
>>>>> PS2 it'd be could to verify tests as well against "bleeding edge"
>>>>> apache version gets installed from SCM, but I don't know how to do
>>>>> this.
>>>>> Currently I use http://httpd.apache.org/download.cgi ( last stable release
)
>>>>>
>>>>>
>>>>>
>>>>> 2016-01-30 21:36 GMT+03:00 Alexey Melezhik <melezhik@gmail.com>:
>>>>>> Hi Bill!
>>>>>>
>>>>>> I have started to assemble swat tests  for apache2 issues coming
from
>>>>>> https://bz.apache.org/bugzilla
>>>>>>
>>>>>> And this the first one - https://bz.apache.org/bugzilla/show_bug.cgi?id=44221
,
>>>>>>
>>>>>> Test suite is failed for the moment :
>>>>>>
>>>>>>
>>>>>> vagrant@Debian-jessie-amd64-netboot:~/my/apache-swat$ swat -t 44221
>>>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.baz/00.GET.t ...
>>>>>> ok 1 - GET 127.0.0.1/44221/foo.baz succeeded
>>>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/52jLAo76YZ.hdr
>>>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/52jLAo76YZ
>>>>>> ok 2 - output match '200 OK'
>>>>>> 1..2
>>>>>> ok
>>>>>> /home/vagrant/.swat/.cache/384/prove/44221/FOO.bar/00.GET.t ...
>>>>>> ok 1 - GET 127.0.0.1/44221/FOO.bar succeeded
>>>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/DvEZnO6ALe.hdr
>>>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/DvEZnO6ALe
>>>>>> ok 2 - output match '200 OK'
>>>>>> ok 3 - output match /Location: \S+/
>>>>>> ok 4 - 'Location: http://127.0.0.1/44221/foo.bar' match 'foo.bar'
>>>>>> 1..4
>>>>>> ok
>>>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.BAR/00.GET.t ...
>>>>>> ok 1 - GET 127.0.0.1/44221/foo.BAR succeeded
>>>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/bWQ1Sbl0YZ.hdr
>>>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/bWQ1Sbl0YZ
>>>>>> ok 2 - output match '200 OK'
>>>>>> ok 3 - output match /Location: \S+/
>>>>>> ok 4 - 'Location: http://127.0.0.1/44221/foo.bar' match 'foo.bar'
>>>>>> 1..4
>>>>>> ok
>>>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.bar/00.GET.t ...
>>>>>> ok 1 - GET 127.0.0.1/44221/foo.bar succeeded
>>>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/Jd_hlo7Qwv.hdr
>>>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/Jd_hlo7Qwv
>>>>>> ok 2 - output match '200 OK'
>>>>>> 1..2
>>>>>> ok
>>>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.html/00.GET.t ..
>>>>>> ok 1 - GET 127.0.0.1/44221/foo.html succeeded
>>>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/tW2L511eym.hdr
>>>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/tW2L511eym
>>>>>> ok 2 - output match /HTTP\/(\S+) (\d+) \S+/
>>>>>> not ok 3 - 'HTTP/1.1 300 Multiple Choices' match /404 /
>>>>>>
>>>>>> #   Failed test ''HTTP/1.1 300 Multiple Choices' match /404 /'
>>>>>> #   at /usr/local/share/perl/5.20.2/swat.pm line 218.
>>>>>> not ok 4 - output match 'Not Found'
>>>>>>
>>>>>> #   Failed test 'output match 'Not Found''
>>>>>> #   at /usr/local/share/perl/5.20.2/swat.pm line 218.
>>>>>> 1..4
>>>>>> # Looks like you failed 2 tests of 4.
>>>>>> Dubious, test returned 2 (wstat 512, 0x200)
>>>>>> Failed 2/4 subtests
>>>>>>
>>>>>> Test Summary Report
>>>>>> -------------------
>>>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.html/00.GET.t (Wstat:
>>>>>> 512 Tests: 4 Failed: 2)
>>>>>> Failed tests:  3-4
>>>>>> Non-zero exit status: 2
>>>>>> Files=5, Tests=16,  0 wallclock secs ( 0.03 usr  0.00 sys +  0.27
cusr
>>>>>> 0.00 csys =  0.30 CPU)
>>>>>> Result: FAIL
>>>>>>
>>>>>> ...
>>>>>>
>>>>>> Ok, this is just the beginning ;-))) , watch soon updates at -
>>>>>> https://github.com/melezhik/apache-swat
>>>>>>
>>>>>> 2016-01-29 20:44 GMT+03:00 William A Rowe Jr <wrowe@rowe-clan.net>:
>>>>>>> On Fri, Jan 29, 2016 at 6:35 AM, Alexey Melezhik <melezhik@gmail.com>
wrote:
>>>>>>>>
>>>>>>>> Hi Bill!
>>>>>>>>
>>>>>>>> Any news? ( Please see my previous reply ...)
>>>>>>>> Intr
>>>>>>>
>>>>>>>
>>>>>>> Intrigued :)  But my responses will be delayed, I personally
>>>>>>> won't have time to look further myself until other backlogged
>>>>>>> commitments to httpd are caught up some more.
>>>>>>>
>>>>>>> Be aware many of us have only a few hours to participate in the
>>>>>>> httpd project itself, so it will take several days or more to
gather
>>>>>>> feedback on your interest and perhaps a champion to drive such
>>>>>>> an effort if it is actively driven from this side.  Of course
that
>>>>>>> should not stop you from assembling some ideas at the vcs of
>>>>>>> your choice, and sharing a link for those who want to look at
an
>>>>>>> early draft of such an effort!
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Bill
>>>>
>

Mime
View raw message