lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: OOM with 2.9
Date Mon, 13 Jul 2009 09:49:41 GMT
> On Sun, Jul 12, 2009 at 10:51 AM, eks dev<eksdev@yahoo.co.uk> wrote:
> 
> > MMapDirectory has support for chunking (Ineteger.MAX_VALUE) anyhow...
> maybe for such cases this threshold can become user settable. I will try
> to experiment with it  (I am talking about MMapDirectory -> private final
> int MAX_BBUF = Integer.MAX_VALUE;)
> 
> This chunking doesn't reduce virtual memory usage, because the buffers
> are all mapped up front on opening the IndexInput.
> 
> It's only done because of a limitation of ByteBuffer (it uses int not
> long to address the values).

The idea of chunking is because of address space fragmentation. If the Java
VM allocates or frees memory (e.g. when expanding the heap from Xms to Xmx),
the address space of the process gets fragmented. If the index is very
large, the byte buffer must fit in complete as one fragment into the address
space, which may easily fail when you only have 32 bit.

If the file is mapped in different chunks, each chunk can be somewhere else
in address space (e.g., not only one 1 Gig junk is needed, there can be ten
100 Meg junks at different places).

Uwe


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message