lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harald Kirsch <harald.kir...@raytion.com>
Subject Re: OutOfMemory when initializing MMapIndexInput on lucene 3.6.2
Date Mon, 18 Aug 2014 11:19:09 GMT
ulimit -v unlimited

might help, see

http://stackoverflow.com/questions/8892143/error-when-opening-a-lucene-index-map-failed

Harald.

On 18.08.2014 13:10, Shlomit Rosen wrote:
> Hi all,
>
> Using lucene 3.6.2, we are trying to search a pretty small collection.
> To open the directory we use Mmap since we are running on a 64 bit linux
> machine, and we usually get much better results than using SimpleFS or
> NIO.
> Although the collection is only a few GB in size, we are getting an OOM
> exception right from the start:
>
> IQQS0003E The searchable object for collection
> /IndexStore/IndexArea/OS2_Document_20140813023606_9970347625CD4483BD700CA08571491C
> cannot be loaded.
> Causes of the problem:
>     IQQS0004E The reader for index
> /IndexStore/IndexArea/OS2_Document_20140813023606_9970347625CD4483BD700CA08571491C/data/text
> cannot be created.
>     IQQG0020E java.io.IOException: Map failed
>     IQQG0020E java.lang.OutOfMemoryError: Map failed
>        sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:784)
>
> org.apache.lucene.store.MMapDirectory$MMapIndexInput.&lt;init&gt;(MMapDirectory.java:270)
>   org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:220)
>
> org.apache.lucene.index.SegmentCoreReaders.&lt;init&gt;(SegmentCoreReaders.java:96)
>        org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
>        org.apache.lucene.index.SegmentReader.get(SegmentReader.java:94)
>
> org.apache.lucene.index.DirectoryReader.&lt;init&gt;(DirectoryReader.java:105)
>
> org.apache.lucene.index.ReadOnlyDirectoryReader.&lt;init&gt;(ReadOnlyDirectoryReader.java:27)
>   org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:78)
>
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:709)
>   org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:72)
>        org.apache.lucene.index.IndexReader.open(IndexReader.java:375)
> ...
>
> Collection details:
>
> File Name      File Size
> ---------      ---------
> _9d.prx        700520240 bytes (668.07 MB)
> _6a.prx        688987818 bytes (657.07 MB)
> _im.prx        678185165 bytes (646.77 MB)
> _37.prx        672422515 bytes (641.27 MB)
> _rv.prx        672034961 bytes (640.90 MB)
> _fj.prx        669692961 bytes (638.67 MB)
> _cg.prx        666485509 bytes (635.61 MB)
> _os.prx        661239341 bytes (630.61 MB)
> _lp.prx        656960684 bytes (626.53 MB)
> _9d.frq        237580704 bytes (226.57 MB)
> _6a.frq        233601024 bytes (222.78 MB)
> _im.frq        229835093 bytes (219.19 MB)
> _37.frq        227920880 bytes (217.36 MB)
> _rv.frq        227857650 bytes (217.30 MB)
> _fj.frq        227030974 bytes (216.51 MB)
> _cg.frq        225955651 bytes (215.49 MB)
> _os.frq        224161119 bytes (213.78 MB)
> _lp.frq        222748045 bytes (212.43 MB)
> _tl.cfs        104218657 bytes (99.39 MB)
> _s2.cfs        99241207 bytes (94.64 MB)
> _ta.cfs        96972944 bytes (92.48 MB)
> _sz.cfs        95444312 bytes (91.02 MB)
> _sd.cfs        95334907 bytes (90.92 MB)
> _so.cfs        94020302 bytes (89.66 MB)
> _9d.fdt        67804161 bytes (64.66 MB)
> _6a.fdt        66670600 bytes (63.58 MB)
> _im.fdt        65635643 bytes (62.60 MB)
> _37.fdt        65082068 bytes (62.07 MB)
> _rv.fdt        65027960 bytes (62.02 MB)
> _fj.fdt        64820141 bytes (61.82 MB)
> _cg.fdt        64519880 bytes (61.53 MB)
> _os.fdt        63995985 bytes (61.03 MB)
> _lp.fdt        63578365 bytes (60.63 MB)
> _ts.cfs        16675689 bytes (15.90 MB)
> _tq.cfs        16630852 bytes (15.86 MB)
> _tn.cfs        10548412 bytes (10.06 MB)
> _to.cfs        9935882 bytes (9.48 MB)
> _tp.cfs        9667762 bytes (9.22 MB)
> _tm.cfs        9473559 bytes (9.03 MB)
> _tk.cfs        8742824 bytes (8.34 MB)
> _9d.tis        8027312 bytes (7.66 MB)
> _6a.tis        7898747 bytes (7.53 MB)
> _im.tis        7775053 bytes (7.41 MB)
> _37.tis        7714470 bytes (7.36 MB)
> _rv.tis        7702330 bytes (7.35 MB)
> _fj.tis        7682979 bytes (7.33 MB)
> _cg.tis        7649182 bytes (7.29 MB)
> _os.tis        7585214 bytes (7.23 MB)
> _lp.tis        7537160 bytes (7.19 MB)
> _tt.cfs        1832496 bytes (1.75 MB)
> _9d.fdx        858540 bytes (838.42 KB)
> _6a.fdx        844204 bytes (824.42 KB)
> _im.fdx        831084 bytes (811.61 KB)
> _37.fdx        824068 bytes (804.75 KB)
> _rv.fdx        823380 bytes (804.08 KB)
> _fj.fdx        820756 bytes (801.52 KB)
> _cg.fdx        816972 bytes (797.82 KB)
> _os.fdx        810332 bytes (791.34 KB)
> _lp.fdx        805044 bytes (786.18 KB)
> _tr.cfs        640731 bytes (625.71 KB)
> _tu.cfs        419728 bytes (409.89 KB)
> _9d.nrm        214638 bytes (209.61 KB)
> _6a.nrm        211054 bytes (206.11 KB)
> _im.nrm        207774 bytes (202.90 KB)
> _37.nrm        206020 bytes (201.19 KB)
> _rv.nrm        205848 bytes (201.02 KB)
> _fj.nrm        205192 bytes (200.38 KB)
> _cg.nrm        204246 bytes (199.46 KB)
> _os.nrm        202586 bytes (197.84 KB)
> _lp.nrm        201264 bytes (196.55 KB)
> _9d.tii        84483 bytes (82.50 KB)
> _6a.tii        83166 bytes (81.22 KB)
> _im.tii        81863 bytes (79.94 KB)
> _37.tii        81277 bytes (79.37 KB)
> _rv.tii        81113 bytes (79.21 KB)
> _fj.tii        80939 bytes (79.04 KB)
> _cg.tii        80709 bytes (78.82 KB)
> _os.tii        79774 bytes (77.90 KB)
> _lp.tii        79369 bytes (77.51 KB)
> segments_qj    5977 bytes (5.84 KB)
> _37.fnm        86 bytes
> _6a.fnm        86 bytes
> _9d.fnm        86 bytes
> _cg.fnm        86 bytes
> _fj.fnm        86 bytes
> _im.fnm        86 bytes
> _lp.fnm        86 bytes
> _os.fnm        86 bytes
> _rv.fnm        86 bytes
>
>
>
> Does this seem like a reasonable behavior or is this a bug?
> It seems to me that with this small segments, memory should not be an
> issue...?
>
>
> We haven't seen this exception in previous testing.
> In the lucene documentation I found that tweaking the chunk size might
> help,
> But this code goes into customer production environment, and this option
> is not feasible.
> Is there anything else I can check ahead of time (except for if this is a
> 32/64 bit platform) to help me decide
> Whether or not Mmap should be used?
>
> Thanks in advance,
> Shlomit
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message