httpd-test-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stas Bekman <s...@stason.org>
Subject Re: mod_include test shuffling
Date Wed, 10 Dec 2003 08:34:43 GMT
Geoffrey Young wrote:
> well, since I know andre is paying attention, I thought I'd start with
> mod_include :)
> 
> ok, I made some adjustments to t/modules/include.t to accommodate 1.3, 2.0,
> and 2.1.  hopefully, all I did was shuffle things around so that it's easier
> to keep track of the three cases: proper behavior, in-process fixes, and
> will-never-be-fixed-in-this-version.  for the most part, these were all
> documented by andre so all I ended up doing was (hopefully) making later
> bookeeping a bit easier for everyone.
> 
> so, here are before and after outputs (note the failure numbers are
> different due to a new ordering):
> 
> 1.3 before:
> modules/include....NOK 18# Failed test 18 in modules/include.t at line 127
> fail #18
> modules/include....NOK 35# Failed test 35 in modules/include.t at line 127
> fail #35
> modules/include....ok 42/76# Failed test 42 in modules/include.t at line 207
> *TODO*
> 
> 1.3 after:
> modules/include....ok 20/76# Failed test 20 in modules/include-geoff.t at
> line 223 fail #18 *TODO*
> modules/include....ok 39/76# Failed test 39 in modules/include-geoff.t at
> line 223 fail #37 *TODO*
> modules/include....ok 43/76# Failed test 43 in modules/include-geoff.t at
> line 272 *TODO*
> modules/include....ok
>         14/76 skipped: Skipping bucket boundary tests, no mod_bucketeer
> All tests successful, 14 subtests skipped.

Just to add to the concern Bill has voiced, there is a risk of "fixing" 
testing here. You see that the sub-tests have failed only if you run in the 
verbose mode. Most people won't do that, and will miss those failures, 
thinking that everything is proper.

I'm +1 your patch and goal, I'm just hoping that those in charge of the tests 
with failures will remember to run in the verbose mode to see those TODO tests.

Actually it's a similar problem with skipped tests, when some component is 
unavailable. What you end up is not running all tests and missing potential 
failures.

__________________________________________________________________
Stas Bekman            JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/     mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org   http://ticketmaster.com


Mime
View raw message