httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fred Moyer <f...@redhotpenguin.com>
Subject Re: [PATCH] New default sort order in Apache-Test-1.36 (WAS: test framework and latest Fedora)
Date Thu, 26 May 2011 00:15:37 GMT
+1

Do you have commit privileges on Apache-Test?

On Fri, May 20, 2011 at 1:51 PM, Rainer Jung <rainer.jung@kippdata.de> wrote:
> I noticed, that the SMOKE change in 1.36 also introduced a new sorting
> behaviour. In the change the test suite was running by default using the
> repeat order, that means the tests were ordered alpabetically.
>
> Now the default order is random.
>
> Furthermore the -order=rotate (A B C A B C) is gone, and -order=repeat
> now behaves like previously -order=rotate (it seems it previously was A
> A B B C C).
>
> I suggest:
>
> - keep behaviour and name of repeat as it is now
> - don't try to reimplement old repeat, because it might be tricky in
> combination with the new way iterations are done (SMOKE)
> - switch back to the new *repeat* (=old rotate) as the default for
> non-SMOKE tests, i.e. run the tests by default in sorted order and only
> do random if -order=random is explicitely given.
> - stay with random as default for SMOKE
>
> The following one line change should do that:
>
> Index: Apache-Test/lib/Apache/TestSort.pm
> ===================================================================
> --- Apache-Test/lib/Apache/TestSort.pm  (revision 1125303)
> +++ Apache-Test/lib/Apache/TestSort.pm  (working copy)
> @@ -54,7 +54,7 @@
>  sub run {
>     my($self, $list, $args) = @_;
>
> -    my $order = $args->{order} || 'random';
> +    my $order = $args->{order} || 'repeat';
>     if ($order =~ /^\d+$/) {
>         #dont want an explicit -seed option but env var can be a pain
>         #so if -order is number assume it is the random seed
>
>
> Furthermore the READMEs and Changelog should get updated:
>
> ´Index: README
> ===================================================================
> --- README      (revision 1125303)
> +++ README      (working copy)
> @@ -77,18 +77,20 @@
>
>  It's also possible that a test will pass when it's run after a
>  particular test, but if moved to run after a different state it may
> -fail. For this reason it's important to ran the randomization smoking
> -testing with -order=random option.
> +fail. For this reason by default the tests run in random order.
>
>  Since it's important to be able to reproduce the problem with the
>  random testing, whenever -order=random is used, the used seed is
>  printed to STDERR. Which can be then fed into the future tests with:
>  via APACHE_TEST_SEED environment variable.
>
> +By adding the option -order=repeat, the tests will be run in
> +alphabetical order.
> +
>  Combining these two important smoke testing techiques, one can run
>  tests with:
>
> -    t/SMOKE -times=N -order=(repeat|rotate|random)
> +    t/SMOKE -times=N -order=(repeat|random)
>
>  For example, to run the mod_rewrite tests 5 times, one would:
>
> @@ -98,8 +100,7 @@
>  modes:
>
>  - randomize all tests
> -- repeat the same test N times in sequence
> -- rotate the whole tests suite N times
> +- repeat the whole tests suite N times
>
>  For configuration options and default settings run:
>
> Index: Apache-Test/README
> ===================================================================
> --- Apache-Test/README  (revision 1125303)
> +++ Apache-Test/README  (working copy)
> @@ -203,18 +203,15 @@
>
>  run all tests 10 times in a random order (the seed is autogenerated
>  and reported)
> -% t/TEST -times=10 -order=random
> +% t/SMOKE -times=10 -order=random
>
>  run all tests 10 times in a random order using the seed obtained from
>  the previous random run (e.g. 2352211):
> -% t/TEST -times=10 -order=2352211
> +% t/SMOKE -times=10 -order=2352211
>
> -rotate all tests 10 times (a, b, c, a, b, c)
> -% t/TEST -times=10 -order=rotate
> +repeat all tests 10 times (a, b, c, a, b, c)
> +% t/SMOKE -times=10 -order=repeat
>
> -repeat all tests 10 times (a, a, b, b, c, c)
> -% t/TEST -times=10 -order=repeat
> -
>  When certain tests fail when running with -times option, you want to
>  find out the minimal sequence of tests that lead to the
>  failure. Apache::TestSmoke helps to ease this task, simply run:
> Index: Apache-Test/Changes
> ===================================================================
> --- Apache-Test/Changes (revision 1125303)
> +++ Apache-Test/Changes (working copy)
> @@ -8,6 +8,9 @@
>
>  =item 1.37-dev
>
> +Run t/TEST tests by default in alphabetical order and
> +only t/SMOKE tests by default in random order.
> +
>  Add t_file_watch_for to Apache::TestUtil [Torsten Foertsch]
>
>  Add $boolean parameter to Apache::TestHandler::ok and
> Apache::TestHandler::ok1
>
> Regards,
>
> Rainer
>

Mime
View raw message