lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: Random testing performance (theoretical)
Date Sun, 24 Jun 2012 15:53:03 GMT
Hi Dawid,

I think we do not do completely random testing at all! Random testing (with the limits mentioned
in the article) would mean that we do very random testing with also randomly generated tests.
In contrast our tests all have a non-random background, means our tests are written with a
specific use-case in mind (there are some exemptions, I agree, see below). So with any random
parameter they basically test what a non-random junit test would also test. In most cases
the randomness is only added on top (to extend the space of parameters we test) - so the chances
to find bugs get greater. If we would test Lucene only with also randomly generated tests,
I agree we would find no bugs at all. On the other hand, classical junit tests have no randomness
at all, they just test that a specific "requirement" is implemented correctly and returns
the correct result. The problem with those tests is the fact, that by fixing a bug to do the
correct thing in test can break something else (this is especially important for IR tests,
they rely on statistics!!!). You could write a test that expects a specific output of a lucene
query and you can of course fix all the scoring logic to exactly create that result. Of course
any other similar query can return bogus results. So by executing random queries, we can also
test statistical things like ranking function works also for a larger set of queries or larger
number of documents.

There are some exceptions: The famous TestRandomChains falls exactly in the category you mention:
They just build a random chain of analyzer components and the number of combinations you could
create is huge! With a few Tokenizers and a list of 150 TokenFilters you can create an insane
amount of TokenStream combinations (just start to multiply..., it is the formula for the binomial
coefficient). In that case, the chance to find a new bug is really small. The number of failures
at the beginning really shows that the TokenStreams were in a very bad state! Now we have
only few failures (one more today!), but I am sure there are still billions (ore more) of
combinations that break the TokenStream contracts!

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: dawid.weiss@gmail.com [mailto:dawid.weiss@gmail.com] On Behalf Of
> Dawid Weiss
> Sent: Sunday, June 24, 2012 3:42 PM
> To: dev@lucene.apache.org
> Cc: idss@cs.put.poznan.pl
> Subject: Random testing performance (theoretical)
> 
> An interesting link was given to me today:
> 
> "The Central Limit Theorem Makes Random Testing Hard"
> http://blog.regehr.org/archives/660
> 
> Just recently I had a talk at a local university
> (http://idss.cs.put.poznan.pl/site/idss-en.html) and the low theoretical
> probability of hitting _any_ sensible bug was also raised as a concern. This is in
> stark contrast to what we see in practice, isn't it? It is an interesting
> phenomenon because it means that (among other possible explanations) the
> parameter/ search space of random tests is just ridden with failures and
> exceptions and bugs and even with close-to-zero probability we still hit a lot of
> them :)
> 
> Dawid
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional
> commands, e-mail: dev-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message