lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Core dumped
Date Fri, 28 May 2010 14:40:07 GMT
Also, are you indexing largish documents?  Lucene must fully index the
doc, and then flush, so for such large docs it can easily use more
than the 50 MB buffer you allotted.

There were some recent memory leak fixes for such large documents, as
well, that you might be hitting.  Which Lucene version are you using?

Finally, a core dump means you've tickled a JRE bug.  Nothing Lucene
(or any java program) does can cause a core dump unless it's a JRE
bug.  That said, it looks quite likely that avoiding the OOME will
sidestep the bug :)

Mike

On Thu, May 27, 2010 at 1:52 PM, Saurabh Agarwal <srbh.grwl@gmail.com> wrote:
> I will check it out!!
> Saurabh Agarwal
>
>
> On Thu, May 27, 2010 at 11:13 PM, Erick Erickson <erickerickson@gmail.com>wrote:
>
>> The larger your RAMbufferSize, the more memory you consume FWIW.
>>
>> OK, then, does it always OOM on the same document? Are you trying to index
>> any particularly large documents?
>>
>> Erick
>>
>> On Thu, May 27, 2010 at 1:28 PM, Saurabh Agarwal <srbh.grwl@gmail.com
>> >wrote:
>>
>> > RAMBufferSize id 50 Mb, i tried with 200 too
>> > the index is unoptimized
>> > MergeFactor is Default 10 and I have not changed it
>> >
>> > MaxBuffered Docs is also default
>> > Saurabh Agarwal
>> >
>> >
>> > On Thu, May 27, 2010 at 10:31 PM, Erick Erickson <
>> erickerickson@gmail.com
>> > >wrote:
>> >
>> > > What have you set various indexwriter properties to? Particularly
>> > > things like merge factor, max buffered docs and ram buffer size.
>> > >
>> > > The first thing I'd look at is MergeFactor. From the JavaDocs:
>> > > Determines how often segment indices are merged by addDocument(). With
>> > > smaller values, less RAM is used while indexing, and searches on
>> > > unoptimized
>> > > indices are faster, but indexing speed is slower. With larger values,
>> > more
>> > > RAM is used during indexing, and while searches on unoptimized indices
>> > are
>> > > slower, indexing is faster. Thus larger values (> 10) are best for
>> batch
>> > > index creation, and smaller values (< 10) for indices that are
>> > > interactively
>> > > maintained.
>> > >
>> > >
>> > > HTH
>> > > Erick
>> > >
>> > > On Thu, May 27, 2010 at 11:52 AM, Saurabh Agarwal <srbh.grwl@gmail.com
>> > > >wrote:
>> > >
>> > > > Hi,
>> > > > when I am running Lucene on a 512 MB system.
>> > > > I am getting the following error
>> > > > Exception in thread "main" java.lang.OutOfMemoryError: Java heap
>> space
>> > > >        at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.lucene.index.DocumentsWriter$ByteBlockAllocator.getByteBlock(DocumentsWriter.java:1206)
>> > > > and sometimes
>> > > >  An unexpected error has been detected by Java Runtime Environment:
>> > > > # Problematic frame:
>> > > > # J  java.nio.ByteBuffer.arrayOffset()I
>> > > >
>> > > >
>> > > > if I decrease my corpus size it indexes it perfectly!!!
>> > > >
>> > > > Can someone tell me how to index a larger corpus even if it takes
>> more
>> > > > time!!!
>> > > >
>> > > > Thanks
>> > > >
>> > > > Saurabh Agarwal
>> > > >
>> > >
>> >
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message