Return-Path: Delivered-To: apmail-lucene-java-dev-archive@www.apache.org Received: (qmail 1285 invoked from network); 4 Feb 2009 13:22:30 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 4 Feb 2009 13:22:30 -0000 Received: (qmail 48498 invoked by uid 500); 4 Feb 2009 13:22:30 -0000 Delivered-To: apmail-lucene-java-dev-archive@lucene.apache.org Received: (qmail 47821 invoked by uid 500); 4 Feb 2009 13:22:28 -0000 Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-dev@lucene.apache.org Delivered-To: mailing list java-dev@lucene.apache.org Received: (qmail 47812 invoked by uid 99); 4 Feb 2009 13:22:28 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Feb 2009 05:22:28 -0800 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [80.190.230.99] (HELO mail.troja.net) (80.190.230.99) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Feb 2009 13:22:18 +0000 Received: from localhost (localhost [127.0.0.1]) by mail.troja.net (Postfix) with ESMTP id 1EEEB4F7F1 for ; Wed, 4 Feb 2009 14:21:57 +0100 (CET) Received: from mail.troja.net ([127.0.0.1]) by localhost (cyca.troja.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 29422-04 for ; Wed, 4 Feb 2009 14:21:55 +0100 (CET) Received: from VEGA (unknown [134.102.249.78]) (using SSLv3 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.troja.net (Postfix) with ESMTP id D60C94F795 for ; Wed, 4 Feb 2009 14:21:54 +0100 (CET) From: "Uwe Schindler" To: References: <9EE76932-E776-4FC5-B1DE-A793E0E77DB6@mikemccandless.com> <6F7E2B94-A937-4E3E-85A5-FFC1E2E11479@mikemccandless.com> Subject: RE: failure in TestTrieRangeQuery Date: Wed, 4 Feb 2009 14:21:54 +0100 Message-ID: <1561FEF8D0A14384A81C17F5F3730192@VEGA> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_NextPart_000_000B_01C986D3.EE2F52B0" X-Mailer: Microsoft Office Outlook 11 In-reply-to: Thread-index: AcmGuBRVSny0cw2ISu+x39SidNb4QwAEoCwA X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.5579 X-Virus-Checked: Checked by ClamAV on apache.org ------=_NextPart_000_000B_01C986D3.EE2F52B0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Hi, atached is a patch for LuceneTestCase and a implementation in TestTrieRangeQuery. It overrides the protected method runTest() and inserts a try-catch around the super.runTest() call. Two new methods newRandom() and newRandom(long) are available for the test case. As each test case is run in an own TestCase object instance (so 5 test methods in a class instantiate 5 instances each method working in separate), the random seed is saved on newRandom() and when the test fails with any Throwable, a message with the seed (if not null) is printed out. If newRandom was never called no message will be printed. This patch has only one problem: If a single test method calls newRandom() more than once, only the last seed is saved and printed out. But each test method in a Testcase should call newRandom() exactly once for usage during the execution of this test method. And it is not thread save (no sync, no volatile), but for tests it's unimportant. Maybe we open a JIRA issue? Uwe ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: uwe@thetaphi.de > -----Original Message----- > From: Michael McCandless [mailto:lucene@mikemccandless.com] > Sent: Wednesday, February 04, 2009 12:02 PM > To: java-dev@lucene.apache.org > Subject: Re: failure in TestTrieRangeQuery > > > Sweet! > > I was wondering (but didn't dig) whether we could extend > LuceneTestCase to expose a getRandom() method (which'd record the > seed), and then override invocation of a test (which I'm not sure > JUnit allows you to do) to add a try/finally that prints out the seeds. > > Though: I thought JUnit invokes tests in the sequential order as they > are defined in your class? (I'm not sure about this... it's just what > seems to be the case). And even if it is the case, it's not clear > that's guaranteed as part of JUnit's "contract". If it isn't, we > could have getRandom take a String name and then on exception we print > out the full name -> seed for all getRandom calls for that test? > > I'd like to to find a simple common API, if we can, so that we can fix > all tests that use Random to use it... though really the mods you had > to make are fairly minimal, so we could simply adopt that per test too. > > Mike > > Uwe Schindler wrote: > > > Hi, > > > >>> : By allowing Random to randomly seed itself, we effectively test a > >>> much > >>> : much larger space, ie every time we all run the test, it's > >>> different. We can > >>> : potentially cast a much larger net than a fixed seed. > >>> > >>> i guess i'm just in favor of less randomness and more iterations. > >>> > >>> : Fixing the bug is the "easy" part; discovering a bug is present is > >>> where > >>> : we need all the help we can get ;) > >>> > >>> yes, but knowing a bug is there w/o having any idea what it is or > >>> how to > >>> trigger it can be very frustrating. > >> > >> I agree, it's frustrating. But I'd prefer to know the bug is there > >> and then > >> writhe in frustration at not being able to reproduce it very easily, > >> then let > >> the bug go undetected. I guess ignorance is not bliss, for me ;) > >> > >>> it would be enough for tests to pick a random number, log it, and > >>> then use > >>> it as the seed ... that way if you get a failure you at least know > >>> what > >>> seed was used and you can then hardcode it temporarily to reproduce/ > >>> debug > >> > >> +1! I like this approach. We could record the seed up front, and > >> then in > >> a try/finally if the test failed, print the seed. > > > > I implemented this for TestTrieRangeQuery (see patch). I catch > > java.lang.Error and print the random seed recorded before (The seed is > > generated by a static Random instance for each test method in > > separate: > > Because you cannot predict the order of tests, each test method > > should have > > its own Random instance). As both the Java 1.4 AssertionError and > > the jUnit > > AssertionFailedError are subclasses of Error, they can be catched and > > rethrown easily. > > > > Uwe > > > test > > .patch > > >--------------------------------------------------------------------- > > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org > > For additional commands, e-mail: java-dev-help@lucene.apache.org > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org > For additional commands, e-mail: java-dev-help@lucene.apache.org ------=_NextPart_000_000B_01C986D3.EE2F52B0 Content-Type: application/octet-stream; name="LuceneTestCase-random.patch" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="LuceneTestCase-random.patch" Index: = contrib/queries/src/test/org/apache/lucene/search/trie/TestTrieRangeQuery= .java =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- = contrib/queries/src/test/org/apache/lucene/search/trie/TestTrieRangeQuery= .java (revision 740444) +++ = contrib/queries/src/test/org/apache/lucene/search/trie/TestTrieRangeQuery= .java (working copy) @@ -36,11 +36,10 @@ {=0A= private static final long distance=3D66666;=0A= =0A= - private static Random rnd=3Dnew Random();=0A= - private static RAMDirectory directory;=0A= - private static IndexSearcher searcher;=0A= + private static final RAMDirectory directory;=0A= + private static final IndexSearcher searcher;=0A= static {=0A= - try {=0A= + try { =0A= directory =3D new RAMDirectory();=0A= IndexWriter writer =3D new IndexWriter(directory, new = WhitespaceAnalyzer(),=0A= true, MaxFieldLength.UNLIMITED);=0A= @@ -136,6 +135,7 @@ }=0A= =0A= private void testRandomTrieAndClassicRangeQuery(final TrieUtils = variant) throws Exception {=0A= + final Random rnd=3DnewRandom();=0A= String field=3D"field"+variant.TRIE_BITS;=0A= // 50 random tests, the tests may also return 0 results, if = min>max, but this is ok=0A= for (int i=3D0; i<50; i++) {=0A= @@ -185,6 +185,7 @@ }=0A= =0A= private void testRangeSplit(final TrieUtils variant) throws Exception = {=0A= + final Random rnd=3DnewRandom();=0A= String field=3D"ascfield"+variant.TRIE_BITS;=0A= // 50 random tests=0A= for (int i=3D0; i<50; i++) {=0A= @@ -225,6 +226,7 @@ }=0A= =0A= private void testSorting(final TrieUtils variant) throws Exception {=0A= + final Random rnd=3DnewRandom();=0A= String field=3D"field"+variant.TRIE_BITS;=0A= // 10 random tests, the index order is ascending,=0A= // so using a reverse sort field should retun descending documents=0A= Index: src/test/org/apache/lucene/util/LuceneTestCase.java =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- src/test/org/apache/lucene/util/LuceneTestCase.java (revision = 740441) +++ src/test/org/apache/lucene/util/LuceneTestCase.java (working copy) @@ -20,6 +20,7 @@ import org.apache.lucene.analysis.TokenStream;=0A= import org.apache.lucene.index.ConcurrentMergeScheduler;=0A= import junit.framework.TestCase;=0A= +import java.util.Random;=0A= =0A= /** Base class for all Lucene unit tests. Currently the=0A= * only added functionality over JUnit's TestCase is=0A= @@ -54,4 +55,41 @@ fail("ConcurrentMergeScheduler hit unhandled exceptions");=0A= }=0A= }=0A= + =0A= + /**=0A= + * Returns a {@link Random} instance for generating random numbers = during the test.=0A= + * The random seed is logged during test execution and printed to = System.out on any failure=0A= + * for reproducing the test using {@link #getRandom(long)} with the = recorded seed=0A= + *.=0A= + */=0A= + public Random newRandom() {=0A= + return newRandom(seedRnd.nextLong());=0A= + }=0A= + =0A= + /**=0A= + * Returns a {@link Random} instance for generating random numbers = during the test.=0A= + * If an error occurs in the test that is not reproducible, you can = use this method to=0A= + * initialize the number generator with the seed that was printed out = during the failing test.=0A= + */=0A= + public Random newRandom(long seed) {=0A= + this.seed=3Dnew Long(seed);=0A= + return new Random(seed);=0A= + }=0A= + =0A= + protected void runTest() throws Throwable {=0A= + try {=0A= + super.runTest();=0A= + } catch (Throwable e) {=0A= + if (seed!=3Dnull) {=0A= + System.out.println("Random seed of test '"+getName()+"' was: = "+seed);=0A= + }=0A= + throw e;=0A= + }=0A= + }=0A= + =0A= + // recorded seed=0A= + protected Long seed=3Dnull;=0A= + =0A= + // static members=0A= + private static final Random seedRnd=3Dnew Random();=0A= }=0A= ------=_NextPart_000_000B_01C986D3.EE2F52B0 Content-Type: text/plain; charset=us-ascii --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org For additional commands, e-mail: java-dev-help@lucene.apache.org ------=_NextPart_000_000B_01C986D3.EE2F52B0--