lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shai Erera (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-4532) TestDirectoryTaxonomyReader.testRefreshReadRecreatedTaxonomy failure
Date Sun, 04 Nov 2012 09:52:12 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-4532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490156#comment-13490156
] 

Shai Erera commented on LUCENE-4532:
------------------------------------

Hmmm ... perhaps this explains the bug and why I cannot reproduce it. Here's a code snippet
from DirTaxoReader.refresh():

{code}
    String t1 = indexReader.getIndexCommit().getUserData().get(DirectoryTaxonomyWriter.INDEX_CREATE_TIME);
    String t2 = r2.getIndexCommit().getUserData().get(DirectoryTaxonomyWriter.INDEX_CREATE_TIME);
    if (t1==null) {
      if (t2!=null) {
        r2.close();
        throw new InconsistentTaxonomyException("Taxonomy was recreated at: "+t2);
      }
    } else if (!t1.equals(t2)) {
      r2.close();
      throw new InconsistentTaxonomyException("Taxonomy was recreated at: "+t2+"  !=  "+t1);
    }
{code}

The code compares the commit data that is known to the indexReader instance at hand, and the
reopened one. If they do not match, it throws an exception, as this TaxoReader cannot be refreshed.

The commit data relies on timestamp, which is a bad thing in general, even if the timestamp
is in nano-second granularity. So what happens if a machine is able to add a category and
commit DirTW in the same nano-second? Or maybe just open DirTW with OpenMode.CREATE and close
it? Or the clocks somehow get messed up? The two timestamps in the commit data may accidentally
be the same, leading to the exception exposed by the stacktrace above.

I tried to modify the test to always use RAMDirectory() and not add any categories, just recreate+close,
but I couldn't reproduce it either. However, I tried that on my laptop, and the timestamp
is nano-second, so it may not be fast enough.

Still, I think that that is a potential bug and may explain the errors that we see. The test
is single threaded, the seed ensures we reproduce everything right, so all is left to blame
is the machine's clock :).

I'll try to verify that a bit more, and on other machines, but I think that we should move
to store an increment version number instead of timestamp.
                
> TestDirectoryTaxonomyReader.testRefreshReadRecreatedTaxonomy failure
> --------------------------------------------------------------------
>
>                 Key: LUCENE-4532
>                 URL: https://issues.apache.org/jira/browse/LUCENE-4532
>             Project: Lucene - Core
>          Issue Type: Bug
>          Components: modules/facet
>            Reporter: Shai Erera
>            Assignee: Shai Erera
>
> The following failure on Jenkins:
> {noformat}
> > Build: http://jenkins.sd-datasolutions.de/job/Lucene-Solr-4.x-Windows/1404/
> > Java: 32bit/jdk1.6.0_37 -client -XX:+UseConcMarkSweepGC
> >
> > 1 tests failed.
> > REGRESSION:  org.apache.lucene.facet.taxonomy.directory.TestDirectoryTaxonomyReader.testRefreshReadRecreatedTaxonomy
> >
> > Error Message:
> >
> >
> > Stack Trace:
> > java.lang.ArrayIndexOutOfBoundsException
> >         at __randomizedtesting.SeedInfo.seed([6AB10D3E4E956CFA:BFB2863DB7E077E0]:0)
> >         at java.lang.System.arraycopy(Native Method)
> >         at org.apache.lucene.facet.taxonomy.directory.ParentArray.refresh(ParentArray.java:99)
> >         at org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyReader.refresh(DirectoryTaxonomyReader.java:407)
> >         at org.apache.lucene.facet.taxonomy.directory.TestDirectoryTaxonomyReader.doTestReadRecreatedTaxono(TestDirectoryTaxonomyReader.java:167)
> >         at org.apache.lucene.facet.taxonomy.directory.TestDirectoryTaxonomyReader.testRefreshReadRecreatedTaxonomy(TestDirectoryTaxonomyReader.java:130)
> >         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >         at java.lang.reflect.Method.invoke(Method.java:597)
> >         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
> >         at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
> >         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
> >         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
> >         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
> >         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
> >         at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
> >         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> >         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> >         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
> >         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> >         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> >         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> >         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
> >         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
> >         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
> >         at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
> >         at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
> >         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
> >         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> >         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> >         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> >         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> >         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> >         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> >         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> >         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> >         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> >         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> >         at java.lang.Thread.run(Thread.java:662)
> >
> >
> >
> >
> > Build Log:
> > [...truncated 5664 lines...]
> > [junit4:junit4] Suite: org.apache.lucene.facet.taxonomy.directory.TestDirectoryTaxonomyReader
> > [junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDirectoryTaxonomyReader
-Dtests.method=testRefreshReadRecreatedTaxonomy -Dtests.seed=6AB10D3E4E956CFA -Dtests.slow=true
-Dtests.locale=fr_CA -Dtests.timezone=Atlantic/Jan_Mayen -Dtests.file.encoding=US-ASCII
> > [junit4:junit4] ERROR   0.06s | TestDirectoryTaxonomyReader.testRefreshReadRecreatedTaxonomy
<<<
> > [junit4:junit4]    > Throwable #1: java.lang.ArrayIndexOutOfBoundsException
> > [junit4:junit4]    >    at __randomizedtesting.SeedInfo.seed([6AB10D3E4E956CFA:BFB2863DB7E077E0]:0)
> > [junit4:junit4]    >    at java.lang.System.arraycopy(Native Method)
> > [junit4:junit4]    >    at org.apache.lucene.facet.taxonomy.directory.ParentArray.refresh(ParentArray.java:99)
> > [junit4:junit4]    >    at org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyReader.refresh(DirectoryTaxonomyReader.java:407)
> > [junit4:junit4]    >    at org.apache.lucene.facet.taxonomy.directory.TestDirectoryTaxonomyReader.doTestReadRecreatedTaxono(TestDirectoryTaxonomyReader.java:167)
> > [junit4:junit4]    >    at org.apache.lucene.facet.taxonomy.directory.TestDirectoryTaxonomyReader.testRefreshReadRecreatedTaxonomy(TestDirectoryTaxonomyReader.java:130)
> > [junit4:junit4]    >    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
> > [junit4:junit4]    >    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > [junit4:junit4]    >    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > [junit4:junit4]    >    at java.lang.reflect.Method.invoke(Method.java:597)
> > [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
> > [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
> > [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
> > [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
> > [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
> > [junit4:junit4]    >    at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
> > [junit4:junit4]    >    at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
> > [junit4:junit4]    >    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> > [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> > [junit4:junit4]    >    at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
> > [junit4:junit4]    >    at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> > [junit4:junit4]    >    at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> > [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> > [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
> > [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
> > [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
> > [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
> > [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
> > [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
> > [junit4:junit4]    >    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> > [junit4:junit4]    >    at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> > [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> > [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> > [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> > [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > [junit4:junit4]    >    at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> > [junit4:junit4]    >    at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> > [junit4:junit4]    >    at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> > [junit4:junit4]    >    at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> > [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> > [junit4:junit4]    >    at java.lang.Thread.run(Thread.java:662)
> > [junit4:junit4]   2> NOTE: test params are: codec=SimpleText, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy):
{}, locale=fr_CA, timezone=Atlantic/Jan_Mayen
> > [junit4:junit4]   2> NOTE: Windows 7 6.1 x86/Sun Microsystems Inc. 1.6.0_37 (32-bit)/cpus=2,threads=1,free=59103720,total=93417472
> > [junit4:junit4]   2> NOTE: All tests run in this JVM: [ObjectToIntMapTest, UnsafeByteArrayInputStreamTest,
IntArrayTest, EncodingTest, CustomAssociationPropertyTest, TestDirectoryTaxonomyWriter, FacetSearchParamsTest,
DefaultEnhancementsIndexingParamsTest, TestCategoryListCache, TestTopKInEachNodeResultHandler,
AdaptiveAccumulatorTest, Vint8Test, TestScoredDocIdCollector, IntToIntMapTest, AssociationPropertyTest,
TestMultiCLExample, ArrayHashMapTest, CategoryListIteratorTest, TestTopKResultsHandlerRandom,
TestTotalFacetCounts, SamplingWrapperTest, TestCharBlockArray, IntToDoubleMapTest, TestFacetsCollector,
TestFacetsAccumulatorWithComplement, SamplingAccumulatorTest, TestTopKResultsHandler, CategoryAttributesIterableTest,
CategoryAttributesStreamTest, TestFacetArrays, TestCompactLabelToOrdinal, CategoryListPayloadStreamTest,
CategoryTokenizerTest, TestAddTaxonomy, TestTotalFacetCountsCache, TestTaxonomyCombined, FacetsPayloadProcessorProviderTest,
OrdinalPolicyTest, PathPolicyTest, TestScoredDocIDsUtils, TestDirectoryTaxonomyReader]
> > [junit4:junit4] Completed in 0.11s, 7 tests, 1 error <<< FAILURES!
> >
> > [...truncated 81 lines...]
> > BUILD FAILED
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\build.xml:335: The following
error occurred while executing this line:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\build.xml:39: The following
error occurred while executing this line:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\lucene\build.xml:519: The
following error occurred while executing this line:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\lucene\common-build.xml:1691:
The following error occurred while executing this line:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\lucene\module-build.xml:61:
The following error occurred while executing this line:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\lucene\common-build.xml:1163:
The following error occurred while executing this line:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\lucene\common-build.xml:827:
There were test failures: 65 suites, 264 tests, 1 error, 2 ignored (1 assumption)
> >
> > Total time: 18 minutes 43 seconds
> > Build step 'Invoke Ant' marked build as failure
> > Archiving artifacts
> > Recording test results
> > Description set: Java: 32bit/jdk1.6.0_37 -client -XX:+UseConcMarkSweepGC
> > Email was triggered for: Failure
> > Sending email for trigger: Failure
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message