Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Message-ID: <193654016.8971270718856832.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 09:27:36 +0000 (UTC)
From: "Michael McCandless (JIRA)" <jira@apache.org>
To: java-dev@lucene.apache.org
Subject: [jira] Commented: (LUCENE-2376) java.lang.OutOfMemoryError:Java
 heap space
In-Reply-To: <837866002.33891270608693521.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/LUCENE-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854876#action_12854876 ] 

Michael McCandless commented on LUCENE-2376:
--------------------------------------------

OK but I suspect the root cause is the same here -- your index seems to have a truly massive number of fields.  Can you post the CheckIndex output?

IW re-uses per-field objects internally, so that many docs with the same field can be indexed more efficiently.  However, when IW sweeps to free up RAM, if it notices an allocated field object hasn't been used recently, because that field name has not occurred in recently added docs, it frees up that memory and logs that "purge field".  So from this output I can see you have at least 43K unique field names.

If you have not disabled norms on these fields you'll certainly run out of memory.  Even if you disable norms, Lucene is in general not optimized for a tremendous number of unique fields and you'll likely hit other issues.


> java.lang.OutOfMemoryError:Java heap space
> ------------------------------------------
>
>                 Key: LUCENE-2376
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2376
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.9.1
>         Environment: Windows
>            Reporter: Shivender Devarakonda
>         Attachments: InfoStreamOutput.txt
>
>
> I see an OutOfMemory error in our product and it is happening when we have some data objects on which we built the index. I see the following OutOfmemory error, this is happening after we call Indexwriter.optimize():
> 4/06/10 02:03:42.160 PM PDT [ERROR] [Lucene Merge Thread #12]  In thread Lucene Merge Thread #12 and the message is org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> 4/06/10 02:03:42.207 PM PDT [VERBOSE] [Lucene Merge Thread #12] [Manager] Uncaught Exception in thread Lucene Merge Thread #12
> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
> 	at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
> 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> 	at java.util.HashMap.resize(HashMap.java:462)
> 	at java.util.HashMap.addEntry(HashMap.java:755)
> 	at java.util.HashMap.put(HashMap.java:385)
> 	at org.apache.lucene.index.FieldInfos.addInternal(FieldInfos.java:256)
> 	at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:366)
> 	at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
> 	at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:116)
> 	at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
> 	at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
> 	at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
> 	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
> 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> 	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
> 4/06/10 02:03:42.895 PM PDT [ERROR]  this writer hit an OutOfMemoryError; cannot complete optimize

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org