lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-1566) Large Lucene index can hit false OOM due to Sun JRE issue
Date Tue, 14 Jul 2009 09:25:14 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730748#action_12730748
] 

Michael McCandless commented on LUCENE-1566:
--------------------------------------------

Alas no simple workaround here.  You could try using the current patch, here (though this
patch will only apply to 2.9, I think).  Or, create your own subclass of FSDir that overrides
the read method of FSIndexInput.

Are you hitting the issue?  Can you provide some details about the size of your index, etc?
 Maybe the output of running "java org.apache.lucene.index.CheckIndex /path/to/index"?

> Large Lucene index can hit false OOM due to Sun JRE issue
> ---------------------------------------------------------
>
>                 Key: LUCENE-1566
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1566
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.4.1
>            Reporter: Michael McCandless
>            Assignee: Simon Willnauer
>            Priority: Minor
>             Fix For: 2.9
>
>         Attachments: LUCENE-1566.patch, LUCENE-1566.patch
>
>
> This is not a Lucene issue, but I want to open this so future google
> diggers can more easily find it.
> There's this nasty bug in Sun's JRE:
>   http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6478546
> The gist seems to be, if you try to read a large (eg 200 MB) number of
> bytes during a single RandomAccessFile.read call, you can incorrectly
> hit OOM.  Lucene does this, with norms, since we read in one byte per
> doc per field with norms, as a contiguous array of length maxDoc().
> The workaround was a custom patch to do large file reads as several
> smaller reads.
> Background here:
>   http://www.nabble.com/problems-with-large-Lucene-index-td22347854.html

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message