lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "308181687" <308181...@qq.com>
Subject Re:   OutOfMemoryError throwed by SimpleMergedSegmentWarmer
Date Sat, 13 Sep 2014 12:30:22 GMT
Hi,  mike
  Is there any config option which let IndexReader to omit all norms and not to load norms
to JVM heap?


Thanks & Best Regards!




------------------ Original ------------------
From:  "lubin";<308181687@qq.com>;
Date:  Sat, Sep 13, 2014 08:20 PM
To:  "java-user"<java-user@lucene.apache.org>; 

Subject:  Re:  OutOfMemoryError throwed by SimpleMergedSegmentWarmer



Hi,  Mike ‍
   In our use case, we have thousands of index fields,  different kind of document have different
 fields.  Do you meant that ‍norms field will consume large memory? Why?‍


   If we decide to disable norms, do we need to rebuild our index entirely? By the way, We
have 8 million documents and our jvm heap is 5G.‍


Thanks & Best Regards!


‍



------------------ Original ------------------
From:  "Michael McCandless";<lucene@mikemccandless.com>;
Date:  Sat, Sep 13, 2014 06:29 PM
To:  "Lucene Users"<java-user@lucene.apache.org>; 

Subject:  Re: OutOfMemoryError throwed by SimpleMergedSegmentWarmer



The warmer just tries to load norms/docValues/etc. for all fields that
have them enabled ... so this is likely telling you an IndexReader
would also hit OOME.

You either need to reduce the number of fields you have indexed, or at
least disable norms (takes 1 byte per doc per indexed field regardless
of whether that doc had indexed that field), or increase HEAP to the
JVM.

Mike McCandless

http://blog.mikemccandless.com


On Sat, Sep 13, 2014 at 4:25 AM, 308181687 <308181687@qq.com> wrote:
> Hi, all
>    we got an OutOfMemoryError throwed ‍by SimpleMergedSegmentWarmer. We use lucene
4.7, and access index file by NRTCachingDirectory/MMapDirectory. Could any body give me
a hand?  Strack trace is as follows:
>
>
>
>
>
>
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java
heap space‍
>      at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
>
>
> Caused by: java.lang.OutOfMemoryError: Java heap space
>         at org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:228)
>         at org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:188)
>         at org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:166)
>         at org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:519)
>         at org.apache.lucene.index.SimpleMergedSegmentWarmer.warm(SimpleMergedSegmentWarmer.java:52)
>         at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4275)
>         at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3743)
>         at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
>         at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)‍
>
>
>
>
> Thanks & Best Regards!

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Mime
  • Unnamed multipart/alternative (inline, 8-Bit, 0 bytes)
View raw message