lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: OOM with 2.9
Date Sun, 12 Jul 2009 12:30:23 GMT
Something other:

 

If you have a tight memory situation and really want to use MMap on 32bit,
you should think about using the new support for the Sun JVM hack to "close"
MMapped regions:
http://hudson.zones.apache.org/hudson/job/Lucene-trunk/javadoc/core/org/apac
he/lucene/store/MMapDirectory.html#setUseUnmap(boolean)

 

You can only use this, if you do not use the system property to switch the
directory implementation. You should instantiate the MMapDirectory directly
and then enable this new feature/hack. As you may be noticed,
FSDirectory.getDirectory()/IndexReader.open(using String/File index
locations) and the correspondingsystem property is deprecated.

 

A additional possibility is to use the new
org.apache.lucene.store.FileSwitchDirectory
(http://hudson.zones.apache.org/hudson/job/Lucene-trunk/javadoc/core/org/apa
che/lucene/store/FileSwitchDirectory.html) and only use MMap for file types,
that are senseful for mapping (opening small and often changing files like
the segments file with MMap is not senseful). MMap should only be used for
index files containing the term dicts and so on. E.g. it is an idea to
disable mmap for stored field files. Note: when using FileSwitchDirectory,
compound files should be disabled.

 

 

-----

Uwe Schindler

H.-H.-Meier-Allee 63, D-28213 Bremen

http://www.thetaphi.de

eMail: uwe@thetaphi.de

 

 

> -----Original Message-----

> From: eks dev [mailto:eksdev@yahoo.co.uk]

> Sent: Sunday, July 12, 2009 1:54 PM

> To: java-user@lucene.apache.org

> Subject: Re: OOM with 2.9

> 

> 

> Thanks Uwe,

> yes it is on 32 bit in extremely tight memory situation. I just was

> curious what the reason is (on bigger machine it works). Lucene core

> requires more real memory? Can you please a bit elaborate, can I estimate

> it somehow? I do not remember seeing any changes that would cause core to

> use more memory (we do not sort, no norms overall... this failed just on

> opening IndexReader)...

> 

> cheers, eks

> 

> 

> 

> 

> ----- Original Message ----

> > From: Uwe Schindler <uwe@thetaphi.de>

> > To: java-user@lucene.apache.org

> > Sent: Sunday, 12 July, 2009 13:39:23

> > Subject: RE: OOM with 2.9

> >

> > Is it a 32 or 64 bit JVM? If 32 bit, this error can happen, if the new

> > Lucene Core needs more real memory and so the MMaped address space is

> > already used. MMapDirectory should only be used with 64 bit machines.

> >

> > -----

> > Uwe Schindler

> > H.-H.-Meier-Allee 63, D-28213 Bremen

> > http://www.thetaphi.de

> > eMail: uwe@thetaphi.de

> >

> > > -----Original Message-----

> > > From: eks dev [mailto:eksdev@yahoo.co.uk]

> > > Sent: Sunday, July 12, 2009 1:24 PM

> > > To: java-user@lucene.apache.org

> > > Subject: Re: OOM with 2.9

> > >

> > >

> > > Stack trace

> > >

> > > java.io.IOException: Map failed

> > >  at sun.nio.ch.FileChannelImpl.map(Unknown Source)

> > >  at org.apache.lucene.store.MMapDirectory$MMapIndexInput.(Unknown

> > > Source)

> > >  at org.apache.lucene.store.MMapDirectory$MMapIndexInput.(Unknown

> > > Source)

> > >  at org.apache.lucene.store.MMapDirectory.openInput(Unknown Source)

> > >  at org.apache.lucene.index.SegmentReader$CoreReaders.(Unknown

> > > Source)

> > >  at org.apache.lucene.index.SegmentReader.get(Unknown Source)

> > >  at org.apache.lucene.index.SegmentReader.get(Unknown Source)

> > >  at org.apache.lucene.index.DirectoryReader.(Unknown Source)

> > >  at org.apache.lucene.index.ReadOnlyDirectoryReader.(Unknown Source)

> > >  at org.apache.lucene.index.DirectoryReader$1.doBody(Unknown Source)

> > >  at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(Unknown

> > > Source)

> > >  at org.apache.lucene.index.DirectoryReader.open(Unknown Source)

> > >  at org.apache.lucene.index.IndexReader.open(Unknown Source)

> > >  at org.apache.lucene.index.IndexReader.open(Unknown Source)

> > >  at

> > >

> com.qspi.su.search.IndexReaderFactory.getNewIndexReaderInstance(IndexReade

> > > rFactory.java:70)

> > >  at com.qspi.su.search.SearchStrategy.(SearchStrategy.java:147)

> > >  at com.qspi.su.search.Searcher.(Searcher.java:170)

> > >  at com.qspi.su.search.Searchers.(Searchers.java:21)

> > >  at com.qspi.su.ipc.SUProtocolImpl.(SUProtocolImpl.java:76)

> > >  at com.qspi.su.ipc.SUServer.run(SUServer.java:84)

> > >  at com.qspi.su.ipc.SUServer.main(SUServer.java:204)

> > > Caused by: java.lang.OutOfMemoryError: Map failed

> > >  at sun.nio.ch.FileChannelImpl.map0(Native Method)

> > >

> > >

> > >

> > >

> > > ---------------------------------------------------------------------

> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org

> > > For additional commands, e-mail: java-user-help@lucene.apache.org

> >

> >

> >

> > ---------------------------------------------------------------------

> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org

> > For additional commands, e-mail: java-user-help@lucene.apache.org

> 

> 

> 

> 

> 

> ---------------------------------------------------------------------

> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org

> For additional commands, e-mail: java-user-help@lucene.apache.org

 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message