lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hoss Man (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-3463) Jenkins trunk tests (nightly only) fail quite often with OOM in Automaton/FST tests
Date Fri, 30 Sep 2011 23:37:45 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118573#comment-13118573
] 

Hoss Man commented on LUCENE-3463:
----------------------------------

bq. There are still some issues with tests that create the index in BeforeClass but cannot
read it later when @UseNoMemoryExpensiveCodec is selected for one test method. Also similar
issues may occur if the test explicitely selects such a codec (Mike found some failing tests,
still investigating).

...i haven't really been following this issue (or looked at the patches) but wouldn't the
most straight forward way to deal with this sort of thing be to support specifying the annotation
that indicate certain codecs are required/prohibited on either the individual test methods
or on the entire test class -- so if a class wants to build the index in an @BeforeClass method,
it has to put the @UseNoMemoryExpensiveCodec or @MustUseCodec("PulsingCodec") on the entire
class and not the individual methods?

(where by "most straight forward" i mean for the test writer, not neccessarily for how it's
implemented in LTC)

In the case where classes are building the indexes in @Before or @Test methods, the codec
anotations could be placed on both the class and the test methods - with the method annotations
overwritting the class annotations...

{code}
/** Most of these tests create indexes too bug for in memory codecs */
@UseNoMemoryExpensiveCodec public class CoolFeatureTest {
  @Before buildIndex() { ... }
  @Test testFooIntensively() { ... }
  @Test testBarIntensively { ... }
  /* this test should work with any codec */
  @Test @UseRandomCodec testSimpleStuff() { ... }
  /* this tests specific to pulsing codec, so it must also use that one */
  @Test @UseCodec("PulsingCodec") testPulsingCodec() { ... }
}
{code}


                
> Jenkins trunk tests (nightly only) fail quite often with OOM in Automaton/FST tests
> -----------------------------------------------------------------------------------
>
>                 Key: LUCENE-3463
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3463
>             Project: Lucene - Java
>          Issue Type: Bug
>    Affects Versions: 4.0
>            Reporter: Uwe Schindler
>             Fix For: 4.0
>
>         Attachments: LUCENE-3463.patch, LUCENE-3463.patch, LUCENE-3463.patch, LUCENE-3463.patch
>
>
> The nightly Job Lucene-trunk quite often fails with OOM (in several methods, not always
in the same test):
> Example from last night (this time a huge Automaton):
> {noformat}
> [junit] java.lang.OutOfMemoryError: Java heap space
> [junit] Dumping heap to /home/hudson/hudson-slave/workspace/Lucene-trunk/heapdumps/java_pid38855.hprof
...
> [junit] Heap dump file created [86965954 bytes in 1.186 secs]
> [junit] Testsuite: org.apache.lucene.index.TestTermsEnum
> [junit] Testcase: testIntersectRandom(org.apache.lucene.index.TestTermsEnum):	Caused
an ERROR
> [junit] Java heap space
> [junit] java.lang.OutOfMemoryError: Java heap space
> [junit] 	at org.apache.lucene.util.automaton.RunAutomaton.<init>(RunAutomaton.java:128)
> [junit] 	at org.apache.lucene.util.automaton.ByteRunAutomaton.<init>(ByteRunAutomaton.java:28)
> [junit] 	at org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:134)
> [junit] 	at org.apache.lucene.index.TestTermsEnum.testIntersectRandom(TestTermsEnum.java:266)
> [junit] 	at org.apache.lucene.util.LuceneTestCase$2$1.evaluate(LuceneTestCase.java:611)
> [junit] 	at org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:148)
> [junit] 	at org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:50)
> [junit] 
> [junit] 
> [junit] Tests run: 6, Failures: 0, Errors: 1, Time elapsed: 11.699 sec
> {noformat}
> Other traces:
> {noformat}
> [junit] Testsuite: org.apache.lucene.util.fst.TestFSTs
> [junit] Testcase: testRealTerms(org.apache.lucene.util.fst.TestFSTs):	Caused an ERROR
> [junit] Java heap space
> [junit] java.lang.OutOfMemoryError: Java heap space
> [junit] 	at org.apache.lucene.util.ArrayUtil.grow(ArrayUtil.java:338)
> [junit] 	at org.apache.lucene.util.fst.FST$BytesWriter.writeBytes(FST.java:927)
> [junit] 	at org.apache.lucene.util.fst.ByteSequenceOutputs.write(ByteSequenceOutputs.java:113)
> [junit] 	at org.apache.lucene.util.fst.ByteSequenceOutputs.write(ByteSequenceOutputs.java:32)
> [junit] 	at org.apache.lucene.util.fst.FST.addNode(FST.java:451)
> [junit] 	at org.apache.lucene.util.fst.NodeHash.add(NodeHash.java:122)
> [junit] 	at org.apache.lucene.util.fst.Builder.compileNode(Builder.java:180)
> [junit] 	at org.apache.lucene.util.fst.Builder.finish(Builder.java:495)
> [junit] 	at org.apache.lucene.index.codecs.memory.MemoryCodec$TermsWriter.finish(MemoryCodec.java:232)
> [junit] 	at org.apache.lucene.index.FreqProxTermsWriterPerField.flush(FreqProxTermsWriterPerField.java:414)
> [junit] 	at org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:92)
> [junit] 	at org.apache.lucene.index.TermsHash.flush(TermsHash.java:117)
> [junit] 	at org.apache.lucene.index.DocInverter.flush(DocInverter.java:80)
> [junit] 	at org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:78)
> [junit] 	at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:472)
> [junit] 	at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:420)
> [junit] 	at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:568)
> [junit] 	at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:366)
> [junit] 	at org.apache.lucene.index.IndexReader.open(IndexReader.java:317)
> [junit] 	at org.apache.lucene.util.fst.TestFSTs.testRealTerms(TestFSTs.java:1034)
> [junit] 	at org.apache.lucene.util.LuceneTestCase$2$1.evaluate(LuceneTestCase.java:611)
> {noformat}
> or:
> {noformat}
> [junit] Testsuite: org.apache.lucene.util.automaton.TestCompiledAutomaton
> [junit] Testcase: testRandom(org.apache.lucene.util.automaton.TestCompiledAutomaton):
Caused an ERROR
> [junit] Java heap space
> [junit] java.lang.OutOfMemoryError: Java heap space
> [junit] 	at org.apache.lucene.util.automaton.RunAutomaton.<init>(RunAutomaton.java:128)
> [junit] 	at org.apache.lucene.util.automaton.ByteRunAutomaton.<init>(ByteRunAutomaton.java:28)
> [junit] 	at org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:134)
> [junit] 	at org.apache.lucene.util.automaton.TestCompiledAutomaton.build(TestCompiledAutomaton.java:39)
> [junit] 	at org.apache.lucene.util.automaton.TestCompiledAutomaton.testTerms(TestCompiledAutomaton.java:55)
> [junit] 	at org.apache.lucene.util.automaton.TestCompiledAutomaton.testRandom(TestCompiledAutomaton.java:101)
> [junit] 	at org.apache.lucene.util.LuceneTestCase$2$1.evaluate(LuceneTestCase.java:611)
> [junit] 	at org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:148)
> [junit] 	at org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:50)
> {noformat}
> Almost every nightly test fails, history: [https://builds.apache.org/job/Lucene-trunk]
> We should maybe raise the max heap space or reduce doc counts/...

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message