apr-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From r..@rkbloom.net
Subject Re: PATCH: apr_reslist_invalidate
Date Mon, 15 Mar 2004 14:37:20 GMT
Quoting Nick Kew <nick@webthing.com>:

> >
> > Personally, I dislike using a random number to simulate a failure rate.
> 
> OK so far.  Perhaps the fact that my degree was maths and my first job
> was in stochastic simulation makes me happier than some with them:-)
> 
> >	The
> > purpose of the test suite isn't to emulate the real world where you need a
> > failure rate.
> 
> I think we'd need something more sophisticated to emulate the real world.
> The difficult case is when all resources on a reslist fail at once.

I don't want to emulate the real world.  I want to watch the code in a
controlled environment.  If we can get full coverage of all edge cases in the
test suite in a controlled environment, then we can be positive that we can
handle anything the real world throws at the code.  In reality, we will never
get 100% coverage, but we can come close.

> 
> >	  All you need is one resource that has failed, so that you can
> > test reslist_invalidate.
> 
> I don't think that's right.  My test generates instances where the number
> of resources drops below the minimum, and we see new resources being
> correctly generated.  A single resource failure won't catch that.

True, but those are two different test cases.  You are using a random number
generator to get test coverage.  This means that each test run is actually
running a slightly different test.

The current test loops through a set number of reslist entries, and generates a
random number to determine if the resource should be considered valid or not. 
What happens if none of the random numbers trigger the invalid case?  You
haven't actually tested the invalidate code path.  I realize that this example
is incredibly unlikely, but it isn't impossible, and it is just my reasoning
taken to extremes.  A less extreme example has the same problem, but it isn't as
pronounced.

In unit testing code, you don't want use a random number generator to simulate
failure (unless you are testing the random number generator).  Instead, you want
a controlled environment where you can re-run the tests over and over again and
you are sure to get the same test every time.  What happens with this test, if
it fails when 100% of the resource are invalid?  The current test could trigger
that failuer on one run, but not the next.  Tracking that down would be almost
impossible.

A better way to write this test is to have a function that loops through the
reslist invalidating a specified number of resources.  Then you can call that
function three or four or five different ways.  Once where it shouldn't
invalidate any resources.  Once where it should invalidate only one resource. 
Once where it invalidates enough resources that exactly min resources remain. 
Once where it invalidates enough so that min - 1 resources remain.  Etcetera.

That way, you are gauranteed to exercise all of the error paths on each run of
the test suite.  Using a random number generator to exercise error paths means
that you _will_ miss some cases that are important to catch.

Now, notice that neither the APR nor the APR-util test suites are good in this
respect.  We do a horrible job at exercising our code properly, which is why
some of our APIs aren't actually portable in APR.  This is pretty much my fault.
 When I created APR initially, I wrote the original test suite, and it worked
well enough for my needs, because I just needed to prove that the code worked. 
I hadn't been converted to the need for unit tests at the time.  I wasn't trying
to fully exercise the code, I was just trying to get enough success that it made
sense to pass the API off to David Reid so that he could port to BeOS.  This
meant that new APIs were added regularly without any tests at all, so that we
are now playing catch-up, and failing at it.  One of the problems is that
writing test cases isn't exactly glamorous work, the other is that most of the
people on this list are software developers, not QA people.  QA people are a
special breed, and having a couple on this list would make our code better.  In
general, software developers (myself included) aren't good at testing code,
because we like to see things work not fail.

> There is another issue where the whole thing fails - e.g. talking to
> a backend that goes down.  We need a testcase to ensure it will fail
> cleanly rather than go into an infinite loop.  If you'd like me to
> attack that one, let me know.

Yeah, that would be great.

Ryan


Mime
View raw message