lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Greg Bowyer <gbow...@fastmail.co.uk>
Subject Re: MappedByteBuffer duplicates
Date Fri, 24 Feb 2017 20:13:17 GMT
You may need to enable this 

https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java#L167

If you are a Sun^H^H^H Oracle JVM.


On Fri, Feb 24, 2017, at 11:31 AM, Kameron Cole wrote:
> Actually, at a certain point, they have crashed the machine. The native 
> file mappings are deallocated (unmapped) by the JVM when the 
> MappedByteBuffers are eligible for garbage collection. The problem we're 
> seeing  is that there are thousands of MappedByteBuffers which are not 
> eligible for garbage collection. The native memory is retained because
> the 
> Lucene code is still referencing the MappedByteBuffer objects on the Java 
> heap. This isn't the fault of Windows or the JVM. It appears to be a
> fault 
> in Lucen, but we can't diagnose it - we can't see why the
> MappedByteBuffer 
> objects are being retained.
> 
> 
> 
> 
> 
> From:   "Uwe Schindler" <uwe@thetaphi.de>
> To:     <java-user@lucene.apache.org>
> Date:   02/24/2017 01:39 PM
> Subject:        RE: MappedByteBuffer duplicates
> 
> 
> 
> Hi,
> 
> that is not an issue, the duplicates are required for so called
> IndexInput 
> clones and splices. Every search request will create many of them. But 
> there is no need to worry, they are just thin wrappers - they don't 
> allocate any extra off-heap memory. They are just there to have a
> separate 
> position(), limit() and other settings for each searcher thread.
> 
> Why do you worry?
> Uwe
> 
> -----
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
> 
> > -----Original Message-----
> > From: Kameron Cole [mailto:kameroncole@us.ibm.com]
> > Sent: Friday, February 24, 2017 7:19 PM
> > To: java-user@lucene.apache.org
> > Subject: MappedByteBuffer duplicates
> > 
> > We have a Lucene engine that creates MappedByteBuffer objects when
> > creating the Lucene index.  I don't know Lucene well enough to know if
> > this standard behavior.
> > 
> > The mapped files are being created by Lucene, via the JRE's NIO APIs
> > native file mapping underneath each MappedByteBuffer object. We see an
> > issue where duplicate MappedByteBuffer objects are being created.  Has
> > anyone seen this?
> > 
> > Thank you!
> > 
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 
> 
> 
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message