lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: OutOfMemory when initializing MMapIndexInput on lucene 3.6.2
Date Thu, 21 Aug 2014 08:27:36 GMT
Hi,

> Thank you very much for the quick reply :)
> 
> Just got a word back from our consumer that setting ulimit -v and ulimit -m to
> unlimited solved there problem.

Great!

> They did raise a question about the best practice for this.
> Assuming a customer will not want to set these flags to unlimit, but just
> increase their value to the minimum required....
> Is there a rule of thumb as to how to calculate the limit size?
> Would it be good enough to set it to the largest segment size of the largest
> collection for example, or something of that effect?

The absolute minimum required for -v would be a maximum virtual address space of the sum of
the size of all indexes that are open at the same time (if they were static). If you do adds/deletes,
you need 2 times of the size of all indexes. And finally, if you do horrible optimize/forceMerge
and have indexes open for read at the same time, you may need 3 times. In addition the virtual
address space gots fragmented over time on dynamic indexes, so not even that is enough.

In general, -v should be unlimited in general! The setting only exists for older 32 bit platforms,
because one process could easily map a large file and consume all available address space,
which is in fact very limited to little more than 2 GiB (because the CPU has no larger pointers).
On 64 bit CPUs, address space is almost free (depends on operating system and processor, in
most cases you can use up to around 44 bits, see blog entry), so the limit makes no real sense
at all. It might useful if you have multiple Lucene/Solr/Elasticsearch instances running on
the same machine, each having indexes like 1 Terabyte of size and you want to prevent that
one index accidently reserves all address space the other index might need to map its files,
too....

-m can be bounded, it has not so much to do with virtual memory, it is about the resident
memory. -m should at least have 1.5 times the heap size of the process (this is what Java
itself allocates) plus some additional memory needed for read/write buffers, system libraries,...
I just mentioned -m here and asked to set it to unlimited, to first try to prevent any problem
that may arise.

If you really need to set some value for -v (I don't recommend) or -m, a good approach would
be to monitor the process with top and set something like VIRT as -v limit and RES as -m limit
(with some buffer, like 1/2 of the common value). In general, limiting -v on 64 bit platforms
is a very bad idea and counteracts the idea behind virtual address space.

Uwe

> Hi,
> 
> For a full description of Lucene & MMap, see:
> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
> 
> Quote: "How to configure my operating system and Java VM to make optimal
> use of MMapDirectory?
> First of all, default settings in Linux distributions and Solaris/Windows are
> perfectly fine. But there are some paranoid system administrators around,
> that want to control everything (with lack of understanding).
> Those limit the maximum amount of virtual address space that can be
> allocated by applications. So please check that ?ulimit -v? and ?ulimit -m?
> both report ?unlimited?, otherwise it may happen that MMapDirectory
> reports ?mmap failed? while opening your index. If this error still happens on
> systems with lot?s of very large indexes, each of those with many segments,
> you may need to tune your kernel parameters in
> /etc/sysctl.conf: The default value of vm.max_map_count is 65530, you may
> need to raise it. I think, for Windows and Solaris systems there are similar
> settings available, but it is up to the reader to find out how to use them.
> 
> For configuring your Java VM, you should rethink your memory
> requirements:
> Give only the really needed amount of heap space and leave as much as
> possible to the O/S. As a rule of thumb: Don?t use more than ΒΌ of your
> physical memory as heap space for Java running Lucene/Solr, keep the
> remaining memory free for the operating system cache. If you have more
> applications running on your server, adjust accordingly. As usual the more
> physical memory the better, but you don?t need as much physical memory
> as your index size. The kernel does a good job in paging in frequently used
> pages from your index.
> 
> A good possibility to check that you have configured your system optimally is
> by looking at both "top" (and correctly interpreting it, see above) and the
> similar command "iotop" (can be installed, e.g., on Ubuntu Linux by "apt-get
> install iotop"). If your system does lots of swap in/swap out for the Lucene
> process, reduce heap size, you possibly used too much. If you see lot's of
> disk I/O, buy more RUM (Simon Willnauer) so mmapped files don't need to
> be paged in/out all the time, and finally: buy SSDs."
> 
> This also has the recommened settings for ulimit and other sysctls.
> Changing the chunk size does not help on 64 bit platforms, it should be 1 GiB
> there.
> 
> Uwe
> 
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
> 
> 
> > -----Original Message-----
> > From: Harald Kirsch [mailto:harald.kirsch@raytion.com]
> > Sent: Monday, August 18, 2014 1:19 PM
> > To: java-user@lucene.apache.org
> > Subject: Re: OutOfMemory when initializing MMapIndexInput on lucene
> > 3.6.2
> >
> > ulimit -v unlimited
> >
> > might help, see
> >
> > http://stackoverflow.com/questions/8892143/error-when-opening-a-
> > lucene-index-map-failed
> >
> > Harald.
> >
> > On 18.08.2014 13:10, Shlomit Rosen wrote:
> > > Hi all,
> > >
> > > Using lucene 3.6.2, we are trying to search a pretty small collection.
> > > To open the directory we use Mmap since we are running on a 64 bit
> > > linux machine, and we usually get much better results than using
> > > SimpleFS or NIO.
> > > Although the collection is only a few GB in size, we are getting an
> > > OOM exception right from the start:
> > >
> > > IQQS0003E The searchable object for collection
> > >
> >
> /IndexStore/IndexArea/OS2_Document_20140813023606_9970347625CD448
> > 3BD70
> > > 0CA08571491C
> > > cannot be loaded.
> > > Causes of the problem:
> > >     IQQS0004E The reader for index
> > >
> >
> /IndexStore/IndexArea/OS2_Document_20140813023606_9970347625CD448
> > 3BD70
> > > 0CA08571491C/data/text
> > > cannot be created.
> > >     IQQG0020E java.io.IOException: Map failed
> > >     IQQG0020E java.lang.OutOfMemoryError: Map failed
> > >        sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:784)
> > >
> > >
> >
> org.apache.lucene.store.MMapDirectory$MMapIndexInput.&lt;init&gt;(MM
> > apDirectory.java:270)
> > >
> > >
> >
> org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:22
> > 0
> > > )
> > >
> > >
> >
> org.apache.lucene.index.SegmentCoreReaders.&lt;init&gt;(SegmentCoreRe
> > aders.java:96)
> > >
> > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
> > >
> > > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:94)
> > >
> > > org.apache.lucene.index.DirectoryReader.&lt;init&gt;(DirectoryReader
> > > .j
> > > ava:105)
> > >
> > >
> > org.apache.lucene.index.ReadOnlyDirectoryReader.&lt;init&gt;(ReadOnlyD
> > ir
> > ectoryReader.java:27)
> > >
> > >
> org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:
> > > 78)
> > >
> > >
> >
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfo
> > s.java:709)
> > > org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:72)
> > >
> > > org.apache.lucene.index.IndexReader.open(IndexReader.java:375)
> > > ...
> > >
> > > Collection details:
> > >
> > > File Name      File Size
> > > ---------      ---------
> > > _9d.prx        700520240 bytes (668.07 MB)
> > > _6a.prx        688987818 bytes (657.07 MB)
> > > _im.prx        678185165 bytes (646.77 MB)
> > > _37.prx        672422515 bytes (641.27 MB)
> > > _rv.prx        672034961 bytes (640.90 MB)
> > > _fj.prx        669692961 bytes (638.67 MB)
> > > _cg.prx        666485509 bytes (635.61 MB)
> > > _os.prx        661239341 bytes (630.61 MB)
> > > _lp.prx        656960684 bytes (626.53 MB)
> > > _9d.frq        237580704 bytes (226.57 MB)
> > > _6a.frq        233601024 bytes (222.78 MB)
> > > _im.frq        229835093 bytes (219.19 MB)
> > > _37.frq        227920880 bytes (217.36 MB)
> > > _rv.frq        227857650 bytes (217.30 MB)
> > > _fj.frq        227030974 bytes (216.51 MB)
> > > _cg.frq        225955651 bytes (215.49 MB)
> > > _os.frq        224161119 bytes (213.78 MB)
> > > _lp.frq        222748045 bytes (212.43 MB)
> > > _tl.cfs        104218657 bytes (99.39 MB)
> > > _s2.cfs        99241207 bytes (94.64 MB)
> > > _ta.cfs        96972944 bytes (92.48 MB)
> > > _sz.cfs        95444312 bytes (91.02 MB)
> > > _sd.cfs        95334907 bytes (90.92 MB)
> > > _so.cfs        94020302 bytes (89.66 MB)
> > > _9d.fdt        67804161 bytes (64.66 MB)
> > > _6a.fdt        66670600 bytes (63.58 MB)
> > > _im.fdt        65635643 bytes (62.60 MB)
> > > _37.fdt        65082068 bytes (62.07 MB)
> > > _rv.fdt        65027960 bytes (62.02 MB)
> > > _fj.fdt        64820141 bytes (61.82 MB)
> > > _cg.fdt        64519880 bytes (61.53 MB)
> > > _os.fdt        63995985 bytes (61.03 MB)
> > > _lp.fdt        63578365 bytes (60.63 MB)
> > > _ts.cfs        16675689 bytes (15.90 MB)
> > > _tq.cfs        16630852 bytes (15.86 MB)
> > > _tn.cfs        10548412 bytes (10.06 MB)
> > > _to.cfs        9935882 bytes (9.48 MB)
> > > _tp.cfs        9667762 bytes (9.22 MB)
> > > _tm.cfs        9473559 bytes (9.03 MB)
> > > _tk.cfs        8742824 bytes (8.34 MB)
> > > _9d.tis        8027312 bytes (7.66 MB)
> > > _6a.tis        7898747 bytes (7.53 MB)
> > > _im.tis        7775053 bytes (7.41 MB)
> > > _37.tis        7714470 bytes (7.36 MB)
> > > _rv.tis        7702330 bytes (7.35 MB)
> > > _fj.tis        7682979 bytes (7.33 MB)
> > > _cg.tis        7649182 bytes (7.29 MB)
> > > _os.tis        7585214 bytes (7.23 MB)
> > > _lp.tis        7537160 bytes (7.19 MB)
> > > _tt.cfs        1832496 bytes (1.75 MB)
> > > _9d.fdx        858540 bytes (838.42 KB)
> > > _6a.fdx        844204 bytes (824.42 KB)
> > > _im.fdx        831084 bytes (811.61 KB)
> > > _37.fdx        824068 bytes (804.75 KB)
> > > _rv.fdx        823380 bytes (804.08 KB)
> > > _fj.fdx        820756 bytes (801.52 KB)
> > > _cg.fdx        816972 bytes (797.82 KB)
> > > _os.fdx        810332 bytes (791.34 KB)
> > > _lp.fdx        805044 bytes (786.18 KB)
> > > _tr.cfs        640731 bytes (625.71 KB)
> > > _tu.cfs        419728 bytes (409.89 KB)
> > > _9d.nrm        214638 bytes (209.61 KB)
> > > _6a.nrm        211054 bytes (206.11 KB)
> > > _im.nrm        207774 bytes (202.90 KB)
> > > _37.nrm        206020 bytes (201.19 KB)
> > > _rv.nrm        205848 bytes (201.02 KB)
> > > _fj.nrm        205192 bytes (200.38 KB)
> > > _cg.nrm        204246 bytes (199.46 KB)
> > > _os.nrm        202586 bytes (197.84 KB)
> > > _lp.nrm        201264 bytes (196.55 KB)
> > > _9d.tii        84483 bytes (82.50 KB)
> > > _6a.tii        83166 bytes (81.22 KB)
> > > _im.tii        81863 bytes (79.94 KB)
> > > _37.tii        81277 bytes (79.37 KB)
> > > _rv.tii        81113 bytes (79.21 KB)
> > > _fj.tii        80939 bytes (79.04 KB)
> > > _cg.tii        80709 bytes (78.82 KB)
> > > _os.tii        79774 bytes (77.90 KB)
> > > _lp.tii        79369 bytes (77.51 KB)
> > > segments_qj    5977 bytes (5.84 KB)
> > > _37.fnm        86 bytes
> > > _6a.fnm        86 bytes
> > > _9d.fnm        86 bytes
> > > _cg.fnm        86 bytes
> > > _fj.fnm        86 bytes
> > > _im.fnm        86 bytes
> > > _lp.fnm        86 bytes
> > > _os.fnm        86 bytes
> > > _rv.fnm        86 bytes
> > >
> > >
> > >
> > > Does this seem like a reasonable behavior or is this a bug?
> > > It seems to me that with this small segments, memory should not be
> > > an issue...?
> > >
> > >
> > > We haven't seen this exception in previous testing.
> > > In the lucene documentation I found that tweaking the chunk size
> > > might help, But this code goes into customer production environment,
> > > and this option is not feasible.
> > > Is there anything else I can check ahead of time (except for if this
> > > is a
> > > 32/64 bit platform) to help me decide Whether or not Mmap should be
> > > used?
> > >
> > > Thanks in advance,
> > > Shlomit
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message