lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Erick Erickson" <erickerick...@gmail.com>
Subject Re: outof memory error
Date Mon, 04 Feb 2008 15:53:04 GMT
ummmm index smaller documents? <G>

You cannot expect to index a 1G doc with 512M of memory in the JVM.
The first thing I'd try is upping your JVM memory to the max your machine
will accept.

Make sure you flush your IndexWriter before attempting to index this
document.

But I would not be surprised if this failed to solve the problem. What's in
this massive document? Would it be possible to break it up into
smaller segments and index many sub-documents for this massive doc?
I also wonder what problem you're trying to solve by indexing this doc.
Is it a log file? I can't imagine a text document that big. That's like a
100 volume encyclopedia, and I can't help but wonder whether your users
would be better served by indexing it in pieces.

Best
Erick

On Feb 4, 2008 10:25 AM, SK R <rsk.sen@gmail.com> wrote:

> Hi,
>   I got outof memory exception while  indexing  huge documents (~1GB) in
> one thread and optimizing some other (2 to 3) indexes in different
> threads.
> Max JVM heap size is 512MB. I'm using lucene2.3.0.
>
>   Please suggest a way to avoid this exception.
>
> Regards
>  RSK
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message