lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-3147) MockDirectoryWrapper should track open file handles of IndexOutput too
Date Sun, 29 May 2011 00:30:47 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040708#comment-13040708
] 

Michael McCandless commented on LUCENE-3147:
--------------------------------------------

I'm still hitting an intermittent fail:
{noformat}
NOTE: reproduce with: ant test -Dtestcase=TestIndexWriterWithThreads -Dtestmethod=testIOExceptionDuringWriteSegmentWithThreads
-Dtests.seed=-3173666621914391759:573268695851991855

java.lang.RuntimeException: MockDirectoryWrapper: cannot close: there are still open files:
{_h.fdt=1, _h.tvd=1, _h.tvf=1, _h.fdx=1, _h.tvx=1}
	at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:454)
	at org.apache.lucene.index.TestIndexWriterWithThreads._testMultipleThreadsFailure(TestIndexWriterWithThreads.java:279)
	at org.apache.lucene.index.TestIndexWriterWithThreads.testIOExceptionDuringWriteSegmentWithThreads(TestIndexWriterWithThreads.java:407)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
	at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
	at org.apache.lucene.util.LuceneTestCase$LuceneTestCaseRunner.runChild(LuceneTestCase.java:1334)
	at org.apache.lucene.util.LuceneTestCase$LuceneTestCaseRunner.runChild(LuceneTestCase.java:1252)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:24)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:136)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:117)
	at org.junit.runner.JUnitCore.runMain(JUnitCore.java:98)
	at org.junit.runner.JUnitCore.runMainAndExit(JUnitCore.java:53)
	at org.junit.runner.JUnitCore.main(JUnitCore.java:45)
Caused by: java.lang.RuntimeException: unclosed IndexOutput: _h.tvf
	at org.apache.lucene.store.MockDirectoryWrapper.addFileHandle(MockDirectoryWrapper.java:393)
	at org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:369)
	at org.apache.lucene.index.TermVectorsTermsWriter.initTermVectorsWriter(TermVectorsTermsWriter.java:98)
	at org.apache.lucene.index.TermVectorsTermsWriter.finishDocument(TermVectorsTermsWriter.java:123)
	at org.apache.lucene.index.TermsHash.finishDocument(TermsHash.java:138)
	at org.apache.lucene.index.DocInverter.finishDocument(DocInverter.java:95)
	at org.apache.lucene.index.DocFieldProcessor.finishDocument(DocFieldProcessor.java:293)
	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:246)
	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:368)
	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1469)
	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1441)
	at org.apache.lucene.index.TestIndexWriterWithThreads$IndexerThread.run(TestIndexWriterWithThreads.java:65)
{noformat}

The seed doesn't repro for me; I have to run in a while(1), but it repros fairly quickly with
-Dtest.iters=10.

I'm not sure how/why but somehow there is a DWPT that successfully indexed 1 doc, opened the
doc stores, and then is never heard from again... when it fails it always still has the 5
doc stores open.  Somehow, when we flushAll at close we fail to visit this DWPT.  I don't
know where it's getting lost...

OK, this has something to do w/ DocsWriterFlushControl.abortFullFlushes -- that method visits
the DWPT in question, to abort its flush, but, this does not close the opened doc store files.

But: why do we ever "abort" a full flush...?  It looks like it happens because one of the
DWPTs hits an exc while flushing...?

> MockDirectoryWrapper should track open file handles of IndexOutput too
> ----------------------------------------------------------------------
>
>                 Key: LUCENE-3147
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3147
>             Project: Lucene - Java
>          Issue Type: Test
>          Components: general/test
>            Reporter: Shai Erera
>            Assignee: Shai Erera
>            Priority: Minor
>             Fix For: 3.2, 4.0
>
>         Attachments: LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch,
LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch, LUCENE-3147.patch
>
>
> MockDirectoryWrapper currently tracks open file handles of IndexInput only. Therefore
IO files that are not closed do not fail our tests, which can then lead to test directories
fail to delete on Windows. We should make sure all open files are tracked and if they are
left open, fail the test. I'll attach a patch shortly.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message