lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shlomit Rosen <SHLOM...@il.ibm.com>
Subject OutOfMemory when initializing MMapIndexInput on lucene 3.6.2
Date Mon, 18 Aug 2014 11:10:59 GMT
Hi all, 

Using lucene 3.6.2, we are trying to search a pretty small collection. 
To open the directory we use Mmap since we are running on a 64 bit linux 
machine, and we usually get much better results than using SimpleFS or 
NIO. 
Although the collection is only a few GB in size, we are getting an OOM 
exception right from the start: 

IQQS0003E The searchable object for collection 
/IndexStore/IndexArea/OS2_Document_20140813023606_9970347625CD4483BD700CA08571491C 
cannot be loaded.
Causes of the problem:
   IQQS0004E The reader for index 
/IndexStore/IndexArea/OS2_Document_20140813023606_9970347625CD4483BD700CA08571491C/data/text

cannot be created.
   IQQG0020E java.io.IOException: Map failed
   IQQG0020E java.lang.OutOfMemoryError: Map failed
      sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:784)
 
org.apache.lucene.store.MMapDirectory$MMapIndexInput.&lt;init&gt;(MMapDirectory.java:270)
 org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:220)
 
org.apache.lucene.index.SegmentCoreReaders.&lt;init&gt;(SegmentCoreReaders.java:96)
      org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
      org.apache.lucene.index.SegmentReader.get(SegmentReader.java:94)
 
org.apache.lucene.index.DirectoryReader.&lt;init&gt;(DirectoryReader.java:105)
 
org.apache.lucene.index.ReadOnlyDirectoryReader.&lt;init&gt;(ReadOnlyDirectoryReader.java:27)
 org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:78)
 
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:709)
 org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:72)
      org.apache.lucene.index.IndexReader.open(IndexReader.java:375)
...

Collection details: 

File Name      File Size
---------      ---------
_9d.prx        700520240 bytes (668.07 MB)
_6a.prx        688987818 bytes (657.07 MB)
_im.prx        678185165 bytes (646.77 MB)
_37.prx        672422515 bytes (641.27 MB)
_rv.prx        672034961 bytes (640.90 MB)
_fj.prx        669692961 bytes (638.67 MB)
_cg.prx        666485509 bytes (635.61 MB)
_os.prx        661239341 bytes (630.61 MB)
_lp.prx        656960684 bytes (626.53 MB)
_9d.frq        237580704 bytes (226.57 MB)
_6a.frq        233601024 bytes (222.78 MB)
_im.frq        229835093 bytes (219.19 MB)
_37.frq        227920880 bytes (217.36 MB)
_rv.frq        227857650 bytes (217.30 MB)
_fj.frq        227030974 bytes (216.51 MB)
_cg.frq        225955651 bytes (215.49 MB)
_os.frq        224161119 bytes (213.78 MB)
_lp.frq        222748045 bytes (212.43 MB)
_tl.cfs        104218657 bytes (99.39 MB)
_s2.cfs        99241207 bytes (94.64 MB)
_ta.cfs        96972944 bytes (92.48 MB)
_sz.cfs        95444312 bytes (91.02 MB)
_sd.cfs        95334907 bytes (90.92 MB)
_so.cfs        94020302 bytes (89.66 MB)
_9d.fdt        67804161 bytes (64.66 MB)
_6a.fdt        66670600 bytes (63.58 MB)
_im.fdt        65635643 bytes (62.60 MB)
_37.fdt        65082068 bytes (62.07 MB)
_rv.fdt        65027960 bytes (62.02 MB)
_fj.fdt        64820141 bytes (61.82 MB)
_cg.fdt        64519880 bytes (61.53 MB)
_os.fdt        63995985 bytes (61.03 MB)
_lp.fdt        63578365 bytes (60.63 MB)
_ts.cfs        16675689 bytes (15.90 MB)
_tq.cfs        16630852 bytes (15.86 MB)
_tn.cfs        10548412 bytes (10.06 MB)
_to.cfs        9935882 bytes (9.48 MB)
_tp.cfs        9667762 bytes (9.22 MB)
_tm.cfs        9473559 bytes (9.03 MB)
_tk.cfs        8742824 bytes (8.34 MB)
_9d.tis        8027312 bytes (7.66 MB)
_6a.tis        7898747 bytes (7.53 MB)
_im.tis        7775053 bytes (7.41 MB)
_37.tis        7714470 bytes (7.36 MB)
_rv.tis        7702330 bytes (7.35 MB)
_fj.tis        7682979 bytes (7.33 MB)
_cg.tis        7649182 bytes (7.29 MB)
_os.tis        7585214 bytes (7.23 MB)
_lp.tis        7537160 bytes (7.19 MB)
_tt.cfs        1832496 bytes (1.75 MB)
_9d.fdx        858540 bytes (838.42 KB)
_6a.fdx        844204 bytes (824.42 KB)
_im.fdx        831084 bytes (811.61 KB)
_37.fdx        824068 bytes (804.75 KB)
_rv.fdx        823380 bytes (804.08 KB)
_fj.fdx        820756 bytes (801.52 KB)
_cg.fdx        816972 bytes (797.82 KB)
_os.fdx        810332 bytes (791.34 KB)
_lp.fdx        805044 bytes (786.18 KB)
_tr.cfs        640731 bytes (625.71 KB)
_tu.cfs        419728 bytes (409.89 KB)
_9d.nrm        214638 bytes (209.61 KB)
_6a.nrm        211054 bytes (206.11 KB)
_im.nrm        207774 bytes (202.90 KB)
_37.nrm        206020 bytes (201.19 KB)
_rv.nrm        205848 bytes (201.02 KB)
_fj.nrm        205192 bytes (200.38 KB)
_cg.nrm        204246 bytes (199.46 KB)
_os.nrm        202586 bytes (197.84 KB)
_lp.nrm        201264 bytes (196.55 KB)
_9d.tii        84483 bytes (82.50 KB)
_6a.tii        83166 bytes (81.22 KB)
_im.tii        81863 bytes (79.94 KB)
_37.tii        81277 bytes (79.37 KB)
_rv.tii        81113 bytes (79.21 KB)
_fj.tii        80939 bytes (79.04 KB)
_cg.tii        80709 bytes (78.82 KB)
_os.tii        79774 bytes (77.90 KB)
_lp.tii        79369 bytes (77.51 KB)
segments_qj    5977 bytes (5.84 KB)
_37.fnm        86 bytes
_6a.fnm        86 bytes
_9d.fnm        86 bytes
_cg.fnm        86 bytes
_fj.fnm        86 bytes
_im.fnm        86 bytes
_lp.fnm        86 bytes
_os.fnm        86 bytes
_rv.fnm        86 bytes



Does this seem like a reasonable behavior or is this a bug? 
It seems to me that with this small segments, memory should not be an 
issue...?


We haven't seen this exception in previous testing. 
In the lucene documentation I found that tweaking the chunk size might 
help, 
But this code goes into customer production environment, and this option 
is not feasible. 
Is there anything else I can check ahead of time (except for if this is a 
32/64 bit platform) to help me decide 
Whether or not Mmap should be used? 

Thanks in advance, 
Shlomit 
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message