Mailing-List: contact dev-help@httpd.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@httpd.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAL7UUk_nycXky=RMTSEBA5cZr7JeezySnZdxY18a_7h9jxKiRw@mail.gmail.com>
References: 
 <CAL7UUk-qyES+aMMSN6aBGi1rb=K3Q+pif4FbU__Lqkku7f2t0w@mail.gmail.com>
	<CACsi253b6=VufVyRW3J0iXYhMGRUzLYY1+uFRENAXfkD+Reaag@mail.gmail.com>
	<CAL7UUk_z-CgL1255w8ScgUW+oFtpuybvzoK_kQykbQNv025N_g@mail.gmail.com>
	<CACsi251twJxjTjEbjEnFuE2x49KX-BrVi=MSAs3-7L=ypaKQ7Q@mail.gmail.com>
	<CAL7UUk_1rvLqgNsOMKN-1GxeBWXb2Q3XVAjbbn4+1th9GdAOcw@mail.gmail.com>
	<CAL7UUk9Mc=MUVaQXPvc+mcaoik+=pFW7EW_ccTXbBmo9W0JB1Q@mail.gmail.com>
	<3A242214-802F-4DAE-A124-7FEB0B23D54D@jaguNET.com>
	<CAL7UUk_nycXky=RMTSEBA5cZr7JeezySnZdxY18a_7h9jxKiRw@mail.gmail.com>
Date: Fri, 5 Feb 2016 17:26:39 +0300
Message-ID: 
 <CAL7UUk-dhWTwwvTDVguG3kTRCVuoqwBEr8ce=RCg3UGU_KzY9Q@mail.gmail.com>
Subject: Re: collaboration request - apache server automation testing with
 swat tool
From: Alexey Melezhik <melezhik@gmail.com>
To: dev@httpd.apache.org
Content-Type: text/plain; charset=UTF-8

> I wonder how possible it would be to transcode the old tests to Swat?

Swat has some architectural specifics. As it simply makes http
requests ( by curl ) and validate server response ( by swat DSL ) it
is considered as black box testing tools , there is no way to interact
with httpd internal structure. So probably most of old tests could be
easily ported but some probably not ( which implies something more
complicated ).

Basically  swat could cover simple and average use cases, everything
that could  expressed in terms of make http requests and then
analyzing an output , see examples for some issues I have expressed in
swat scenarios at https://github.com/melezhik/apache-swat


Also alternatively we may run 2 frameworks in parallel. Apache-swat is
intended for developers usage first and is able to run per-issues
test.

This how I see possible  workflow:

* developer accept a bug request
* developer ( or some else ) create swat test for given bug/feature
* so we have `test first` approach - everyone involved in bug/feature
resolution has possibility to verify changes by simply running :

 swat -t $issue-id


2016-02-05 17:08 GMT+03:00 Alexey Melezhik <melezhik@gmail.com>:
> 2016-02-05 17:01 GMT+03:00 Jim Jagielski <jim@jagunet.com>:
>> Personally, I like the idea of having another framework;
>> the current one is OK but somewhat "painful" to update.
>
> Hi! What kind of possible issues you here ? when saying "painful" to
> update, please explain.
> Thanks.
>
> Also , please consider this link - https://github.com/melezhik/apache-swat and
> this post - http://mail-archives.apache.org/mod_mbox/httpd-dev/201602.mbox/%3CCAL7UUk-zqb1=7wDvCe8E-Ku=U=aY0MdJ_-JgCjVzZdvAranKQg@mail.gmail.com%3E
> as latest version of this idea ....
>
>
>
>
>
>>
>> I wonder how possible it would be to transcode the old tests
>> to Swat? We could then provide for 2 testing frameworks, one
>> developed by the ASF and the other external and 3rd party.
>>
>>
>>> On Feb 1, 2016, at 5:23 AM, Alexey Melezhik <melezhik@gmail.com> wrote:
>>>
>>> Here the list of existed issues I was able to automate tests for:
>>>
>>> vagrant@Debian-jessie-amd64-netboot:~/my/apache-swat$ ls -1 | grep -P '\d'
>>> 44221
>>> 46751
>>> 58789
>>> 58828
>>> 58854
>>>
>>>
>>> I have informed developers at https://bz.apache.org/bugzilla/ about
>>> these, so they rely upon them. Or you have some pre-release testing,
>>> so run all of these in a whole chunk?
>>>
>>> Regards
>>>
>>> Alexey
>>>
>>> PS chaining issues / tests listing  as always -
>>> https://github.com/melezhik/apache-swat
>>>
>>> PS2 it'd be could to verify tests as well against "bleeding edge"
>>> apache version gets installed from SCM, but I don't know how to do
>>> this.
>>> Currently I use http://httpd.apache.org/download.cgi ( last stable release )
>>>
>>>
>>>
>>> 2016-01-30 21:36 GMT+03:00 Alexey Melezhik <melezhik@gmail.com>:
>>>> Hi Bill!
>>>>
>>>> I have started to assemble swat tests  for apache2 issues coming from
>>>> https://bz.apache.org/bugzilla
>>>>
>>>> And this the first one - https://bz.apache.org/bugzilla/show_bug.cgi?id=44221 ,
>>>>
>>>> Test suite is failed for the moment :
>>>>
>>>>
>>>> vagrant@Debian-jessie-amd64-netboot:~/my/apache-swat$ swat -t 44221
>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.baz/00.GET.t ...
>>>> ok 1 - GET 127.0.0.1/44221/foo.baz succeeded
>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/52jLAo76YZ.hdr
>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/52jLAo76YZ
>>>> ok 2 - output match '200 OK'
>>>> 1..2
>>>> ok
>>>> /home/vagrant/.swat/.cache/384/prove/44221/FOO.bar/00.GET.t ...
>>>> ok 1 - GET 127.0.0.1/44221/FOO.bar succeeded
>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/DvEZnO6ALe.hdr
>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/DvEZnO6ALe
>>>> ok 2 - output match '200 OK'
>>>> ok 3 - output match /Location: \S+/
>>>> ok 4 - 'Location: http://127.0.0.1/44221/foo.bar' match 'foo.bar'
>>>> 1..4
>>>> ok
>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.BAR/00.GET.t ...
>>>> ok 1 - GET 127.0.0.1/44221/foo.BAR succeeded
>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/bWQ1Sbl0YZ.hdr
>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/bWQ1Sbl0YZ
>>>> ok 2 - output match '200 OK'
>>>> ok 3 - output match /Location: \S+/
>>>> ok 4 - 'Location: http://127.0.0.1/44221/foo.bar' match 'foo.bar'
>>>> 1..4
>>>> ok
>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.bar/00.GET.t ...
>>>> ok 1 - GET 127.0.0.1/44221/foo.bar succeeded
>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/Jd_hlo7Qwv.hdr
>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/Jd_hlo7Qwv
>>>> ok 2 - output match '200 OK'
>>>> 1..2
>>>> ok
>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.html/00.GET.t ..
>>>> ok 1 - GET 127.0.0.1/44221/foo.html succeeded
>>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/tW2L511eym.hdr
>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/tW2L511eym
>>>> ok 2 - output match /HTTP\/(\S+) (\d+) \S+/
>>>> not ok 3 - 'HTTP/1.1 300 Multiple Choices' match /404 /
>>>>
>>>> #   Failed test ''HTTP/1.1 300 Multiple Choices' match /404 /'
>>>> #   at /usr/local/share/perl/5.20.2/swat.pm line 218.
>>>> not ok 4 - output match 'Not Found'
>>>>
>>>> #   Failed test 'output match 'Not Found''
>>>> #   at /usr/local/share/perl/5.20.2/swat.pm line 218.
>>>> 1..4
>>>> # Looks like you failed 2 tests of 4.
>>>> Dubious, test returned 2 (wstat 512, 0x200)
>>>> Failed 2/4 subtests
>>>>
>>>> Test Summary Report
>>>> -------------------
>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.html/00.GET.t (Wstat:
>>>> 512 Tests: 4 Failed: 2)
>>>>  Failed tests:  3-4
>>>>  Non-zero exit status: 2
>>>> Files=5, Tests=16,  0 wallclock secs ( 0.03 usr  0.00 sys +  0.27 cusr
>>>> 0.00 csys =  0.30 CPU)
>>>> Result: FAIL
>>>>
>>>> ...
>>>>
>>>> Ok, this is just the beginning ;-))) , watch soon updates at -
>>>> https://github.com/melezhik/apache-swat
>>>>
>>>> 2016-01-29 20:44 GMT+03:00 William A Rowe Jr <wrowe@rowe-clan.net>:
>>>>> On Fri, Jan 29, 2016 at 6:35 AM, Alexey Melezhik <melezhik@gmail.com> wrote:
>>>>>>
>>>>>> Hi Bill!
>>>>>>
>>>>>> Any news? ( Please see my previous reply ...)
>>>>>> Intr
>>>>>
>>>>>
>>>>> Intrigued :)  But my responses will be delayed, I personally
>>>>> won't have time to look further myself until other backlogged
>>>>> commitments to httpd are caught up some more.
>>>>>
>>>>> Be aware many of us have only a few hours to participate in the
>>>>> httpd project itself, so it will take several days or more to gather
>>>>> feedback on your interest and perhaps a champion to drive such
>>>>> an effort if it is actively driven from this side.  Of course that
>>>>> should not stop you from assembling some ideas at the vcs of
>>>>> your choice, and sharing a link for those who want to look at an
>>>>> early draft of such an effort!
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Bill
>>