lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: OutOfMemory when initializing MMapIndexInput on lucene 3.6.2
Date Mon, 18 Aug 2014 11:49:32 GMT
Hi,

For a full description of Lucene & MMap, see:
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html

Quote: "How to configure my operating system and Java VM to make optimal use of MMapDirectory?
First of all, default settings in Linux distributions and Solaris/Windows are perfectly fine.
But there are some paranoid system administrators around, that want to control everything
(with lack of understanding). Those limit the maximum amount of virtual address space that
can be allocated by applications. So please check that “ulimit -v” and “ulimit -m”
both report “unlimited”, otherwise it may happen that MMapDirectory reports “mmap failed”
while opening your index. If this error still happens on systems with lot’s of very large
indexes, each of those with many segments, you may need to tune your kernel parameters in
/etc/sysctl.conf: The default value of vm.max_map_count is 65530, you may need to raise it.
I think, for Windows and Solaris systems there are similar settings available, but it is up
to the reader to find out how to use them.

For configuring your Java VM, you should rethink your memory requirements: Give only the really
needed amount of heap space and leave as much as possible to the O/S. As a rule of thumb:
Don’t use more than ¼ of your physical memory as heap space for Java running Lucene/Solr,
keep the remaining memory free for the operating system cache. If you have more applications
running on your server, adjust accordingly. As usual the more physical memory the better,
but you don’t need as much physical memory as your index size. The kernel does a good job
in paging in frequently used pages from your index.

A good possibility to check that you have configured your system optimally is by looking at
both "top" (and correctly interpreting it, see above) and the similar command "iotop" (can
be installed, e.g., on Ubuntu Linux by "apt-get install iotop"). If your system does lots
of swap in/swap out for the Lucene process, reduce heap size, you possibly used too much.
If you see lot's of disk I/O, buy more RUM (Simon Willnauer) so mmapped files don't need to
be paged in/out all the time, and finally: buy SSDs."

This also has the recommened settings for ulimit and other sysctls. Changing the chunk size
does not help on 64 bit platforms, it should be 1 GiB there.

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: Harald Kirsch [mailto:harald.kirsch@raytion.com]
> Sent: Monday, August 18, 2014 1:19 PM
> To: java-user@lucene.apache.org
> Subject: Re: OutOfMemory when initializing MMapIndexInput on lucene
> 3.6.2
> 
> ulimit -v unlimited
> 
> might help, see
> 
> http://stackoverflow.com/questions/8892143/error-when-opening-a-
> lucene-index-map-failed
> 
> Harald.
> 
> On 18.08.2014 13:10, Shlomit Rosen wrote:
> > Hi all,
> >
> > Using lucene 3.6.2, we are trying to search a pretty small collection.
> > To open the directory we use Mmap since we are running on a 64 bit
> > linux machine, and we usually get much better results than using
> > SimpleFS or NIO.
> > Although the collection is only a few GB in size, we are getting an
> > OOM exception right from the start:
> >
> > IQQS0003E The searchable object for collection
> >
> /IndexStore/IndexArea/OS2_Document_20140813023606_9970347625CD448
> 3BD70
> > 0CA08571491C
> > cannot be loaded.
> > Causes of the problem:
> >     IQQS0004E The reader for index
> >
> /IndexStore/IndexArea/OS2_Document_20140813023606_9970347625CD448
> 3BD70
> > 0CA08571491C/data/text
> > cannot be created.
> >     IQQG0020E java.io.IOException: Map failed
> >     IQQG0020E java.lang.OutOfMemoryError: Map failed
> >        sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:784)
> >
> >
> org.apache.lucene.store.MMapDirectory$MMapIndexInput.&lt;init&gt;(MM
> apDirectory.java:270)
> >
> >
> org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:22
> 0
> > )
> >
> >
> org.apache.lucene.index.SegmentCoreReaders.&lt;init&gt;(SegmentCoreRe
> aders.java:96)
> >
> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
> >
> > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:94)
> >
> > org.apache.lucene.index.DirectoryReader.&lt;init&gt;(DirectoryReader.j
> > ava:105)
> >
> >
> org.apache.lucene.index.ReadOnlyDirectoryReader.&lt;init&gt;(ReadOnlyDir
> ectoryReader.java:27)
> >
> > org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:
> > 78)
> >
> >
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfo
> s.java:709)
> >   org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:72)
> >        org.apache.lucene.index.IndexReader.open(IndexReader.java:375)
> > ...
> >
> > Collection details:
> >
> > File Name      File Size
> > ---------      ---------
> > _9d.prx        700520240 bytes (668.07 MB)
> > _6a.prx        688987818 bytes (657.07 MB)
> > _im.prx        678185165 bytes (646.77 MB)
> > _37.prx        672422515 bytes (641.27 MB)
> > _rv.prx        672034961 bytes (640.90 MB)
> > _fj.prx        669692961 bytes (638.67 MB)
> > _cg.prx        666485509 bytes (635.61 MB)
> > _os.prx        661239341 bytes (630.61 MB)
> > _lp.prx        656960684 bytes (626.53 MB)
> > _9d.frq        237580704 bytes (226.57 MB)
> > _6a.frq        233601024 bytes (222.78 MB)
> > _im.frq        229835093 bytes (219.19 MB)
> > _37.frq        227920880 bytes (217.36 MB)
> > _rv.frq        227857650 bytes (217.30 MB)
> > _fj.frq        227030974 bytes (216.51 MB)
> > _cg.frq        225955651 bytes (215.49 MB)
> > _os.frq        224161119 bytes (213.78 MB)
> > _lp.frq        222748045 bytes (212.43 MB)
> > _tl.cfs        104218657 bytes (99.39 MB)
> > _s2.cfs        99241207 bytes (94.64 MB)
> > _ta.cfs        96972944 bytes (92.48 MB)
> > _sz.cfs        95444312 bytes (91.02 MB)
> > _sd.cfs        95334907 bytes (90.92 MB)
> > _so.cfs        94020302 bytes (89.66 MB)
> > _9d.fdt        67804161 bytes (64.66 MB)
> > _6a.fdt        66670600 bytes (63.58 MB)
> > _im.fdt        65635643 bytes (62.60 MB)
> > _37.fdt        65082068 bytes (62.07 MB)
> > _rv.fdt        65027960 bytes (62.02 MB)
> > _fj.fdt        64820141 bytes (61.82 MB)
> > _cg.fdt        64519880 bytes (61.53 MB)
> > _os.fdt        63995985 bytes (61.03 MB)
> > _lp.fdt        63578365 bytes (60.63 MB)
> > _ts.cfs        16675689 bytes (15.90 MB)
> > _tq.cfs        16630852 bytes (15.86 MB)
> > _tn.cfs        10548412 bytes (10.06 MB)
> > _to.cfs        9935882 bytes (9.48 MB)
> > _tp.cfs        9667762 bytes (9.22 MB)
> > _tm.cfs        9473559 bytes (9.03 MB)
> > _tk.cfs        8742824 bytes (8.34 MB)
> > _9d.tis        8027312 bytes (7.66 MB)
> > _6a.tis        7898747 bytes (7.53 MB)
> > _im.tis        7775053 bytes (7.41 MB)
> > _37.tis        7714470 bytes (7.36 MB)
> > _rv.tis        7702330 bytes (7.35 MB)
> > _fj.tis        7682979 bytes (7.33 MB)
> > _cg.tis        7649182 bytes (7.29 MB)
> > _os.tis        7585214 bytes (7.23 MB)
> > _lp.tis        7537160 bytes (7.19 MB)
> > _tt.cfs        1832496 bytes (1.75 MB)
> > _9d.fdx        858540 bytes (838.42 KB)
> > _6a.fdx        844204 bytes (824.42 KB)
> > _im.fdx        831084 bytes (811.61 KB)
> > _37.fdx        824068 bytes (804.75 KB)
> > _rv.fdx        823380 bytes (804.08 KB)
> > _fj.fdx        820756 bytes (801.52 KB)
> > _cg.fdx        816972 bytes (797.82 KB)
> > _os.fdx        810332 bytes (791.34 KB)
> > _lp.fdx        805044 bytes (786.18 KB)
> > _tr.cfs        640731 bytes (625.71 KB)
> > _tu.cfs        419728 bytes (409.89 KB)
> > _9d.nrm        214638 bytes (209.61 KB)
> > _6a.nrm        211054 bytes (206.11 KB)
> > _im.nrm        207774 bytes (202.90 KB)
> > _37.nrm        206020 bytes (201.19 KB)
> > _rv.nrm        205848 bytes (201.02 KB)
> > _fj.nrm        205192 bytes (200.38 KB)
> > _cg.nrm        204246 bytes (199.46 KB)
> > _os.nrm        202586 bytes (197.84 KB)
> > _lp.nrm        201264 bytes (196.55 KB)
> > _9d.tii        84483 bytes (82.50 KB)
> > _6a.tii        83166 bytes (81.22 KB)
> > _im.tii        81863 bytes (79.94 KB)
> > _37.tii        81277 bytes (79.37 KB)
> > _rv.tii        81113 bytes (79.21 KB)
> > _fj.tii        80939 bytes (79.04 KB)
> > _cg.tii        80709 bytes (78.82 KB)
> > _os.tii        79774 bytes (77.90 KB)
> > _lp.tii        79369 bytes (77.51 KB)
> > segments_qj    5977 bytes (5.84 KB)
> > _37.fnm        86 bytes
> > _6a.fnm        86 bytes
> > _9d.fnm        86 bytes
> > _cg.fnm        86 bytes
> > _fj.fnm        86 bytes
> > _im.fnm        86 bytes
> > _lp.fnm        86 bytes
> > _os.fnm        86 bytes
> > _rv.fnm        86 bytes
> >
> >
> >
> > Does this seem like a reasonable behavior or is this a bug?
> > It seems to me that with this small segments, memory should not be an
> > issue...?
> >
> >
> > We haven't seen this exception in previous testing.
> > In the lucene documentation I found that tweaking the chunk size might
> > help, But this code goes into customer production environment, and
> > this option is not feasible.
> > Is there anything else I can check ahead of time (except for if this
> > is a
> > 32/64 bit platform) to help me decide
> > Whether or not Mmap should be used?
> >
> > Thanks in advance,
> > Shlomit
> >
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message