lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-2376) java.lang.OutOfMemoryError:Java heap space
Date Thu, 08 Apr 2010 09:27:36 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854876#action_12854876
] 

Michael McCandless commented on LUCENE-2376:
--------------------------------------------

OK but I suspect the root cause is the same here -- your index seems to have a truly massive
number of fields.  Can you post the CheckIndex output?

IW re-uses per-field objects internally, so that many docs with the same field can be indexed
more efficiently.  However, when IW sweeps to free up RAM, if it notices an allocated field
object hasn't been used recently, because that field name has not occurred in recently added
docs, it frees up that memory and logs that "purge field".  So from this output I can see
you have at least 43K unique field names.

If you have not disabled norms on these fields you'll certainly run out of memory.  Even if
you disable norms, Lucene is in general not optimized for a tremendous number of unique fields
and you'll likely hit other issues.


> java.lang.OutOfMemoryError:Java heap space
> ------------------------------------------
>
>                 Key: LUCENE-2376
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2376
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.9.1
>         Environment: Windows
>            Reporter: Shivender Devarakonda
>         Attachments: InfoStreamOutput.txt
>
>
> I see an OutOfMemory error in our product and it is happening when we have some data
objects on which we built the index. I see the following OutOfmemory error, this is happening
after we call Indexwriter.optimize():
> 4/06/10 02:03:42.160 PM PDT [ERROR] [Lucene Merge Thread #12]  In thread Lucene Merge
Thread #12 and the message is org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError:
Java heap space
> 4/06/10 02:03:42.207 PM PDT [VERBOSE] [Lucene Merge Thread #12] [Manager] Uncaught Exception
in thread Lucene Merge Thread #12
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java
heap space
> 	at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> 	at java.util.HashMap.resize(HashMap.java:462)
> 	at java.util.HashMap.addEntry(HashMap.java:755)
> 	at java.util.HashMap.put(HashMap.java:385)
> 	at org.apache.lucene.index.FieldInfos.addInternal(FieldInfos.java:256)
> 	at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:366)
> 	at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> 	at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> 	at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> 	at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> 	at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> 	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> 	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
> 4/06/10 02:03:42.895 PM PDT [ERROR]  this writer hit an OutOfMemoryError; cannot complete
optimize

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message