lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Clemens Wyss DEV <clemens...@mysign.ch>
Subject AW: Issue with Lucene 3.6.1 and MMapDirectory
Date Wed, 14 May 2014 17:52:50 GMT
Not closing an IndexReader most probably (to say the least) results in a mem-leak -> OOM

> But if I close if given that it is share by multiple threads I will 
>need to check each time before doing the search if IndexReader is still open correct?
You can make use of IndexReader#incRef/#decRef , i.e.
ir.incRef();
try
{
<your search code>
}
finally
{
ir.decRef();
}
...
IFF ir.getRefCount() > 1 THEN you are safe to close the "old" ir.

Maybe  SearcherManager http://blog.mikemccandless.com/2011/09/lucenes-searchermanager-simplifies.html
 fits your needs?


-----Urspr√ľngliche Nachricht-----
Von: Liviu Matei [mailto:liviumat@gmail.com] 
Gesendet: Mittwoch, 14. Mai 2014 11:06
An: java-user@lucene.apache.org
Betreff: Issue with Lucene 3.6.1 and MMapDirectory

Hi,

I am encountering the following issue with Lucene 3.6.1 if you could let me know if I am doing
something wrong / there is a mistake I am making it would be great.

In order to improve the performance of the application that I am working at I went to the
approach of reusing the IndexReader and reopening it every 8 hours in order to get the latest
changes. (IndexReader is declared as a global static variable). The search method is called
from multiple threads in parallel so the index reader is shared between threads. Now if I
don't close the old index reader I am noticing increases of virtual memory with every new
reindex reopen (this should not be an issue on 64 bit Linux correct - this is the configuration
I am using and the indexes are on a shared mount NTFS file system ).
   Also from time to times I noticed JVM crasches with the following stack:
 Thread Stack Trace:
    at memcpy+160()@0x381aa7b060
    -- Java stack --
    at java/nio/DirectByteBuffer.get(DirectByteBuffer.java:294)[optimized]
    at
org/apache/lucene/store/MMapDirectory$MMapIndexInput.readBytes(MMapDirectory.java:298)[optimized]
    at org/apache/lucene/store/DataInput.readBytes(DataInput.java:72)
    at
org/apache/lucene/index/CompoundFileReader$CSIndexInput.readInternal(CompoundFileReader.java:275)[optimized]
    at
org/apache/lucene/store/BufferedIndexInput.refill(BufferedIndexInput.java:270)[optimized]
    at
org/apache/lucene/store/BufferedIndexInput.readByte(BufferedIndexInput.java:40)[inlined]
    at
org/apache/lucene/store/DataInput.readVInt(DataInput.java:107)[inlined]
    at
org/apache/lucene/store/BufferedIndexInput.readVInt(BufferedIndexInput.java:217)[optimized]
    at org/apache/lucene/index/FieldsReader.doc(FieldsReader.java:235)
    at
org/apache/lucene/index/SegmentReader.document(SegmentReader.java:492)
    at
org/apache/lucene/index/DirectoryReader.document(DirectoryReader.java:568)
    at org/apache/lucene/index/MultiReader.document(MultiReader.java:252)
    at org/apache/lucene/index/IndexReader.document(IndexReader.java:1138)
    at
org/apache/lucene/search/IndexSearcher.doc(IndexSearcher.java:258)[inlined]


Can you please tell me if all this corruption is caused by the fact that I am not closing
the old IndexReader. But if I close if given that it is share by multiple threads I will need
to check each time before doing the search if IndexReader is still open correct? Let's say
in a thread I am reopening the IndexReader and in another thread I am afterwards reusing the
old one in that case I should do the check correct? Or is there a smarter mechanism in place.

Any help with this would be more than welcome.


Thank you very much and best regards,
Liviu
Mime
View raw message