lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: failure in TestTrieRangeQuery
Date Wed, 04 Feb 2009 13:26:11 GMT
I forgot to mention: If a test fails, the message using the seed is printed
to stdout. The developer can then change the test temporarily:

LuceneTestCase.newRandom() -> LuceneTestCase.newRandom(long)

using the seed from the failed test printout.

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Wednesday, February 04, 2009 2:22 PM
> To: java-dev@lucene.apache.org
> Subject: RE: failure in TestTrieRangeQuery
> 
> Hi,
> 
> atached is a patch for LuceneTestCase and a implementation in
> TestTrieRangeQuery. It overrides the protected method runTest() and
> inserts
> a try-catch around the super.runTest() call. Two new methods newRandom()
> and
> newRandom(long) are available for the test case. As each test case is run
> in
> an own TestCase object instance (so 5 test methods in a class instantiate
> 5
> instances each method working in separate), the random seed is saved on
> newRandom() and when the test fails with any Throwable, a message with the
> seed (if not null) is printed out. If newRandom was never called no
> message
> will be printed.
> 
> This patch has only one problem: If a single test method calls newRandom()
> more than once, only the last seed is saved and printed out. But each test
> method in a Testcase should call newRandom() exactly once for usage during
> the execution of this test method. And it is not thread save (no sync, no
> volatile), but for tests it's unimportant.
> 
> Maybe we open a JIRA issue?
> 
> Uwe
> 
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
> 
> > -----Original Message-----
> > From: Michael McCandless [mailto:lucene@mikemccandless.com]
> > Sent: Wednesday, February 04, 2009 12:02 PM
> > To: java-dev@lucene.apache.org
> > Subject: Re: failure in TestTrieRangeQuery
> >
> >
> > Sweet!
> >
> > I was wondering (but didn't dig) whether we could extend
> > LuceneTestCase to expose a getRandom() method (which'd record the
> > seed), and then override invocation of a test (which I'm not sure
> > JUnit allows you to do) to add a try/finally that prints out the seeds.
> >
> > Though: I thought JUnit invokes tests in the sequential order as they
> > are defined in your class?  (I'm not sure about this... it's just what
> > seems to be the case).  And even if it is the case, it's not clear
> > that's guaranteed as part of JUnit's "contract".  If it isn't, we
> > could have getRandom take a String name and then on exception we print
> > out the full name -> seed for all getRandom calls for that test?
> >
> > I'd like to to find a simple common API, if we can, so that we can fix
> > all tests that use Random to use it... though really the mods you had
> > to make are fairly minimal, so we could simply adopt that per test too.
> >
> > Mike
> >
> > Uwe Schindler wrote:
> >
> > > Hi,
> > >
> > >>> : By allowing Random to randomly seed itself, we effectively test a
> > >>> much
> > >>> : much larger space, ie every time we all run the test, it's
> > >>> different.  We can
> > >>> : potentially cast a much larger net than a fixed seed.
> > >>>
> > >>> i guess i'm just in favor of less randomness and more iterations.
> > >>>
> > >>> : Fixing the bug is the "easy" part; discovering a bug is present is
> > >>> where
> > >>> : we need all the help we can get ;)
> > >>>
> > >>> yes, but knowing a bug is there w/o having any idea what it is or
> > >>> how to
> > >>> trigger it can be very frustrating.
> > >>
> > >> I agree, it's frustrating.  But I'd prefer to know the bug is there
> > >> and then
> > >> writhe in frustration at not being able to reproduce it very easily,
> > >> then let
> > >> the bug go undetected.  I guess ignorance is not bliss, for me ;)
> > >>
> > >>> it would be enough for tests to pick a random number, log it, and
> > >>> then use
> > >>> it as the seed ... that way if you get a failure you at least know
> > >>> what
> > >>> seed was used and you can then hardcode it temporarily to reproduce/
> > >>> debug
> > >>
> > >> +1!  I like this approach.  We could record the seed up front, and
> > >> then in
> > >>  a try/finally if the test failed, print the seed.
> > >
> > > I implemented this for TestTrieRangeQuery (see patch). I catch
> > > java.lang.Error and print the random seed recorded before (The seed is
> > > generated by a static Random instance for each test method in
> > > separate:
> > > Because you cannot predict the order of tests, each test method
> > > should have
> > > its own Random instance). As both the Java 1.4 AssertionError and
> > > the jUnit
> > > AssertionFailedError are subclasses of Error, they can be catched and
> > > rethrown easily.
> > >
> > > Uwe
> > > <random-trie-
> > > test
> > > .patch
> > > >---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-dev-help@lucene.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-dev-help@lucene.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message