lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-2376) java.lang.OutOfMemoryError:Java heap space
Date Wed, 07 Apr 2010 09:26:33 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854398#action_12854398
] 

Michael McCandless commented on LUCENE-2376:
--------------------------------------------

Is this the same issue as LUCENE-2361?  Or is something different here?  Are you using the
same settings in IndexWriter?

The infoStream output is now much more complete -- eg I see where the OOME happens at the
end.

It does look like the cause here is an insane number of fields:

{code}
  purge field=Cor414D5120514D475231202020202020204A94834B21C22E6B
  purge field=Cor414D5120514D475231202020202020204A94834B21C22E6C
  purge field=Cor414D5120514D475231202020202020204A94834B21C22E6D
  purge field=Cor414D5120514D475231202020202020204A94834B21C22E6E
  purge field=Cor414D5120514D475231202020202020204A94834B21C22E70
  purge field=Cor414D5120514D475231202020202020204A94834B21C22E72
{code}

I count 43K purge fields lines in your log (and that's an undercount since you've pruned some).
 Are you disabling norms for these fields (if not, this will certainly lead to OOME)?  In
your design, is there any limit to the number of fields?  Or will each document have a new
field name?

Can you run CheckIndex on the index and post the output?

> java.lang.OutOfMemoryError:Java heap space
> ------------------------------------------
>
>                 Key: LUCENE-2376
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2376
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.9.1
>         Environment: Windows
>            Reporter: Shivender Devarakonda
>         Attachments: InfoStreamOutput.txt
>
>
> I see an OutOfMemory error in our product and it is happening when we have some data
objects on which we built the index. I see the following OutOfmemory error, this is happening
after we call Indexwriter.optimize():
> 4/06/10 02:03:42.160 PM PDT [ERROR] [Lucene Merge Thread #12]  In thread Lucene Merge
Thread #12 and the message is org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError:
Java heap space
> 4/06/10 02:03:42.207 PM PDT [VERBOSE] [Lucene Merge Thread #12] [Manager] Uncaught Exception
in thread Lucene Merge Thread #12
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java
heap space
> 	at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> 	at java.util.HashMap.resize(HashMap.java:462)
> 	at java.util.HashMap.addEntry(HashMap.java:755)
> 	at java.util.HashMap.put(HashMap.java:385)
> 	at org.apache.lucene.index.FieldInfos.addInternal(FieldInfos.java:256)
> 	at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:366)
> 	at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> 	at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> 	at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> 	at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> 	at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> 	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> 	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
> 4/06/10 02:03:42.895 PM PDT [ERROR]  this writer hit an OutOfMemoryError; cannot complete
optimize

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message