httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rainer Jung <rainer.j...@kippdata.de>
Subject Re: [PATCH] New default sort order in Apache-Test-1.36 (WAS: test framework and latest Fedora)
Date Thu, 26 May 2011 07:50:07 GMT
Hi Fred,

On 26.05.2011 02:15, Fred Moyer wrote:
> +1
> 
> Do you have commit privileges on Apache-Test?

Surprise, yes :)

Committed as r1127812.

Regards,

Rainer

> On Fri, May 20, 2011 at 1:51 PM, Rainer Jung <rainer.jung@kippdata.de> wrote:
>> I noticed, that the SMOKE change in 1.36 also introduced a new sorting
>> behaviour. In the change the test suite was running by default using the
>> repeat order, that means the tests were ordered alpabetically.
>>
>> Now the default order is random.
>>
>> Furthermore the -order=rotate (A B C A B C) is gone, and -order=repeat
>> now behaves like previously -order=rotate (it seems it previously was A
>> A B B C C).
>>
>> I suggest:
>>
>> - keep behaviour and name of repeat as it is now
>> - don't try to reimplement old repeat, because it might be tricky in
>> combination with the new way iterations are done (SMOKE)
>> - switch back to the new *repeat* (=old rotate) as the default for
>> non-SMOKE tests, i.e. run the tests by default in sorted order and only
>> do random if -order=random is explicitely given.
>> - stay with random as default for SMOKE
>>
>> The following one line change should do that:
>>
>> Index: Apache-Test/lib/Apache/TestSort.pm
>> ===================================================================
>> --- Apache-Test/lib/Apache/TestSort.pm  (revision 1125303)
>> +++ Apache-Test/lib/Apache/TestSort.pm  (working copy)
>> @@ -54,7 +54,7 @@
>>  sub run {
>>     my($self, $list, $args) = @_;
>>
>> -    my $order = $args->{order} || 'random';
>> +    my $order = $args->{order} || 'repeat';
>>     if ($order =~ /^\d+$/) {
>>         #dont want an explicit -seed option but env var can be a pain
>>         #so if -order is number assume it is the random seed
>>
>>
>> Furthermore the READMEs and Changelog should get updated:
>>
>> ┬┤Index: README
>> ===================================================================
>> --- README      (revision 1125303)
>> +++ README      (working copy)
>> @@ -77,18 +77,20 @@
>>
>>  It's also possible that a test will pass when it's run after a
>>  particular test, but if moved to run after a different state it may
>> -fail. For this reason it's important to ran the randomization smoking
>> -testing with -order=random option.
>> +fail. For this reason by default the tests run in random order.
>>
>>  Since it's important to be able to reproduce the problem with the
>>  random testing, whenever -order=random is used, the used seed is
>>  printed to STDERR. Which can be then fed into the future tests with:
>>  via APACHE_TEST_SEED environment variable.
>>
>> +By adding the option -order=repeat, the tests will be run in
>> +alphabetical order.
>> +
>>  Combining these two important smoke testing techiques, one can run
>>  tests with:
>>
>> -    t/SMOKE -times=N -order=(repeat|rotate|random)
>> +    t/SMOKE -times=N -order=(repeat|random)
>>
>>  For example, to run the mod_rewrite tests 5 times, one would:
>>
>> @@ -98,8 +100,7 @@
>>  modes:
>>
>>  - randomize all tests
>> -- repeat the same test N times in sequence
>> -- rotate the whole tests suite N times
>> +- repeat the whole tests suite N times
>>
>>  For configuration options and default settings run:
>>
>> Index: Apache-Test/README
>> ===================================================================
>> --- Apache-Test/README  (revision 1125303)
>> +++ Apache-Test/README  (working copy)
>> @@ -203,18 +203,15 @@
>>
>>  run all tests 10 times in a random order (the seed is autogenerated
>>  and reported)
>> -% t/TEST -times=10 -order=random
>> +% t/SMOKE -times=10 -order=random
>>
>>  run all tests 10 times in a random order using the seed obtained from
>>  the previous random run (e.g. 2352211):
>> -% t/TEST -times=10 -order=2352211
>> +% t/SMOKE -times=10 -order=2352211
>>
>> -rotate all tests 10 times (a, b, c, a, b, c)
>> -% t/TEST -times=10 -order=rotate
>> +repeat all tests 10 times (a, b, c, a, b, c)
>> +% t/SMOKE -times=10 -order=repeat
>>
>> -repeat all tests 10 times (a, a, b, b, c, c)
>> -% t/TEST -times=10 -order=repeat
>> -
>>  When certain tests fail when running with -times option, you want to
>>  find out the minimal sequence of tests that lead to the
>>  failure. Apache::TestSmoke helps to ease this task, simply run:
>> Index: Apache-Test/Changes
>> ===================================================================
>> --- Apache-Test/Changes (revision 1125303)
>> +++ Apache-Test/Changes (working copy)
>> @@ -8,6 +8,9 @@
>>
>>  =item 1.37-dev
>>
>> +Run t/TEST tests by default in alphabetical order and
>> +only t/SMOKE tests by default in random order.
>> +
>>  Add t_file_watch_for to Apache::TestUtil [Torsten Foertsch]
>>
>>  Add $boolean parameter to Apache::TestHandler::ok and
>> Apache::TestHandler::ok1
>>
>> Regards,
>>
>> Rainer

Mime
View raw message