lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Core dumped
Date Thu, 27 May 2010 17:43:09 GMT
The larger your RAMbufferSize, the more memory you consume FWIW.

OK, then, does it always OOM on the same document? Are you trying to index
any particularly large documents?

Erick

On Thu, May 27, 2010 at 1:28 PM, Saurabh Agarwal <srbh.grwl@gmail.com>wrote:

> RAMBufferSize id 50 Mb, i tried with 200 too
> the index is unoptimized
> MergeFactor is Default 10 and I have not changed it
>
> MaxBuffered Docs is also default
> Saurabh Agarwal
>
>
> On Thu, May 27, 2010 at 10:31 PM, Erick Erickson <erickerickson@gmail.com
> >wrote:
>
> > What have you set various indexwriter properties to? Particularly
> > things like merge factor, max buffered docs and ram buffer size.
> >
> > The first thing I'd look at is MergeFactor. From the JavaDocs:
> > Determines how often segment indices are merged by addDocument(). With
> > smaller values, less RAM is used while indexing, and searches on
> > unoptimized
> > indices are faster, but indexing speed is slower. With larger values,
> more
> > RAM is used during indexing, and while searches on unoptimized indices
> are
> > slower, indexing is faster. Thus larger values (> 10) are best for batch
> > index creation, and smaller values (< 10) for indices that are
> > interactively
> > maintained.
> >
> >
> > HTH
> > Erick
> >
> > On Thu, May 27, 2010 at 11:52 AM, Saurabh Agarwal <srbh.grwl@gmail.com
> > >wrote:
> >
> > > Hi,
> > > when I am running Lucene on a 512 MB system.
> > > I am getting the following error
> > > Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> > >        at
> > >
> > >
> >
> org.apache.lucene.index.DocumentsWriter$ByteBlockAllocator.getByteBlock(DocumentsWriter.java:1206)
> > > and sometimes
> > >  An unexpected error has been detected by Java Runtime Environment:
> > > # Problematic frame:
> > > # J  java.nio.ByteBuffer.arrayOffset()I
> > >
> > >
> > > if I decrease my corpus size it indexes it perfectly!!!
> > >
> > > Can someone tell me how to index a larger corpus even if it takes more
> > > time!!!
> > >
> > > Thanks
> > >
> > > Saurabh Agarwal
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message