httpd-test-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stas Bekman <>
Subject Re: Skipped tests marked as failed?
Date Mon, 12 Nov 2001 04:48:37 GMT
I've updated the section on skipping sub-tests, you can find it in the 
modperl-docs/src/devel/writing_tests/writing_tests.pod, but here is the 
relevant part:

=head2 Skipping Sub-tests

If the standard output line contains the substring I< # Skip> (with
variations in spacing and case) after I<ok> or I<ok NUMBER>, it is
counted as a skipped test. C<Test::Harness> reports the text after I<
# Skip\S*\s+> as a reason for skipping. So you can count a sub-test as
a skipped as follows:

   print "ok 3 # Skip for some reason\n";

or using the C<Apache::Test>'s skip() function which works similarly
to ok():

   skip $should_skip, $test_me;

so if C<$should_skip> is true, the test will be reported as
skipped. The second argument is the one that's sent to ok(), so if
C<$should_skip> is true, a normal ok() sub-test is run. The following
example represent four possible outcomes of using the skip() function:

   use Apache::Test;
   plan tests => 4;

   my $ok     = 1;
   my $not_ok = 0;

   my $should_skip = "foo is missing";
   skip $should_skip, $ok;
   skip $should_skip, $not_ok;

   $should_skip = '';
   skip $should_skip, $ok;
   skip $should_skip, $not_ok;

now we run the test:

   % ./t/TEST -run -v skip_subtest_1
   ok 1 # skip foo is missing
   ok 2 # skip foo is missing
   ok 3
   not ok 4
   # Failed test 4 in skip_subtest_1.t at line 13
   Failed 1/1 test scripts, 0.00% okay. 1/4 subtests failed, 75.00% okay.

As you can see since C<$should_skip> had a true value, the first two
sub-tests were explicitly skipped (using C<$should_skip> as a reason),
so the second argument to skip didn't matter. In the last two
sub-tests C<$should_skip> had a false value therefore the second
argument was passed to the ok() function. Basically the following

   $should_skip = '';
   skip $should_skip, $ok;
   skip $should_skip, $not_ok;

is equivalent to:

   ok $ok;
   ok $not_ok;

C<Apache::Test> also allows to write tests in such a way that only
selected sub-tests will be run.  The test simply needs to switch from
using ok() to sok().  Where the argument to sok() is a CODE reference
or a BLOCK whose return value will be passed to ok().  If sub-tests
are specified on the command line only those will be run/passed to
ok(), the rest will be skipped.  If no sub-tests are specified, sok()
works just like ok().  For example, you can write this test:

   use Apache::Test;
   plan tests => 4;
   sok {1};
   sok {0};
   sok sub {'true'};
   sok sub {''};

and then ask to run only sub-tests 1 and 3 and to skip the rest.

   % ./t/TEST -v skip_subtest_2 1 3
   ok 1
   ok 2 # skip skipping this subtest
   ok 3
   ok 4 # skip skipping this subtest
   ok, 2/4 skipped:  skipping this subtest
   All tests successful, 2 subtests skipped.

Only the sub-tests 1 and 3 get executed.

A range of sub-tests to run can be given using the Perl's range

   % ./t/TEST -v skip_subtest_2 2..4
   ok 1 # skip askipping this subtest
   not ok 2
   # Failed test 2
   ok 3
   not ok 4
   # Failed test 4
   Failed 1/1 test scripts, 0.00% okay. 2/4 subtests failed, 50.00% okay.

In this run, only the first sub-test gets executed.
Stas Bekman             JAm_pH      --   Just Another mod_perl Hacker      mod_perl Guide

View raw message