lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From eks dev <eks...@yahoo.co.uk>
Subject Re: OOM with 2.9
Date Mon, 13 Jul 2009 11:11:54 GMT

Hi Mike, 

thanks for looking into it... 

I am now positive, it was definitely a problem for OS to map() large continuous chunk of process
memory... if I use this machine for a while as a desktop, eclipse,... I get the same problem
again... but after cold restart, mapping succeeds.

The problem never happened with 2.4, but I guess this was a pure coincidence. 

Cheers, 
eks




----- Original Message ----
> From: Michael McCandless <lucene@mikemccandless.com>
> To: java-user@lucene.apache.org
> Sent: Monday, 13 July, 2009 11:52:31
> Subject: Re: OOM with 2.9
> 
> Ahh good point.  I agree it makes sense to make MMapDir's chunking
> user-controllable.  Can you open an issue?
> 
> Mike
> 
> On Mon, Jul 13, 2009 at 5:49 AM, Uwe Schindlerwrote:
> >> On Sun, Jul 12, 2009 at 10:51 AM, eks devwrote:
> >>
> >> > MMapDirectory has support for chunking (Ineteger.MAX_VALUE) anyhow...
> >> maybe for such cases this threshold can become user settable. I will try
> >> to experiment with it  (I am talking about MMapDirectory -> private final
> >> int MAX_BBUF = Integer.MAX_VALUE;)
> >>
> >> This chunking doesn't reduce virtual memory usage, because the buffers
> >> are all mapped up front on opening the IndexInput.
> >>
> >> It's only done because of a limitation of ByteBuffer (it uses int not
> >> long to address the values).
> >
> > The idea of chunking is because of address space fragmentation. If the Java
> > VM allocates or frees memory (e.g. when expanding the heap from Xms to Xmx),
> > the address space of the process gets fragmented. If the index is very
> > large, the byte buffer must fit in complete as one fragment into the address
> > space, which may easily fail when you only have 32 bit.
> >
> > If the file is mapped in different chunks, each chunk can be somewhere else
> > in address space (e.g., not only one 1 Gig junk is needed, there can be ten
> > 100 Meg junks at different places).
> >
> > Uwe
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org



      

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message