lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Lucene not closing an IndexOutput when writing?
Date Wed, 09 Dec 2015 11:26:37 GMT
Hi Trejkaz,

Lucene holds this file handle open for a long time, since term vectors
are appended "live" to this file as the documents are indexed, v.s.
e.g. postings where all buffering is in RAM until it's time to write a
segment at which point we open the output, write stuff, close it.

That said, Lucene tries hard to close this file handle, e.g. if an
in-memory segment is aborted because of e.g. an interrupt exception at
a bad time.

So, yes, please try to make a test showing that we failed to close it!
 That's a bad bug if so ... we should never leak file handles.

Mike McCandless

http://blog.mikemccandless.com


On Tue, Dec 8, 2015 at 7:36 PM, Trejkaz <trejkaz@trypticon.org> wrote:
> We have a Directory implementation that keeps track of who doesn't
> close their IndexInput and IndexOutput.
>
> In some test which is attempting to index documents and ultimately
> timed out for other reasons (presumably triggering an interrupt,
> admittedly not the sort of thing libraries are usually very good at
> handling), it records that the following stack opened an IndexOutput
> but didn't close it:
>
>     java.lang.RuntimeException: unclosed IndexOutput: _0.tvx
>         at com.acme.storage.textindex.store.CloseCheckingDirectory.addOpenResource(CloseCheckingDirectory.java:82)
>         at com.acme.storage.textindex.store.CloseCheckingDirectory.createOutput(CloseCheckingDirectory.java:68)
>         at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
>         at org.apache.lucene.codecs.compressing.CompressingTermVectorsWriter.<init>(CompressingTermVectorsWriter.java:224)
>         at org.apache.lucene.codecs.compressing.CompressingTermVectorsFormat.vectorsWriter(CompressingTermVectorsFormat.java:98)
>         at org.apache.lucene.index.TermVectorsConsumer.initTermVectorsWriter(TermVectorsConsumer.java:88)
>         at org.apache.lucene.index.TermVectorsConsumer.finishDocument(TermVectorsConsumer.java:103)
>         at org.apache.lucene.index.TermsHash.finishDocument(TermsHash.java:93)
>         at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:316)
>         at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:232)
>         at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:458)
>         at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1363)
>         at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1142)
>
> This is in v5.2.1.
>
> I don't know if this is worth a bug record yet, but I thought I would
> ask where this is *supposed* to be closed, so that I can try to make a
> more direct test and maybe catch it in the act.
>
> TX
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message