lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Document larger than setRAMBufferSizeMB()
Date Mon, 06 Oct 2008 15:06:04 GMT

OK it sounds like you need to increase the RAM your JVM is allowed to  
use, or, make your documents smaller.

Mike

Aditi Goyal wrote:

> Thanks for showing interest Mike.
> The OOME comes in the middle of setting a value of one of the field  
> in the
> doc. That field has a fairly large value. May be that could have  
> been the
> reason.?
>
>
>
> On Fri, Oct 3, 2008 at 4:57 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>>
>> Note that large stored fields do not use up any RAM in  
>> IndexWriter's RAM
>> buffer because these stored fields are immediately written to the  
>> directory
>> and not stored in RAM for very long.
>>
>> Aditi, I'd love to see the full stack trace of the OOME that was  
>> originally
>> hit if you still have it...
>>
>> Mike
>>
>>
>> Ganesh wrote:
>>
>> Single document of 16 MB seems to be big. I think you are trying to  
>> store
>>> the entire document content. If it is so drop the stored field and  
>>> store its
>>> reference information in the database, which could help to  
>>> retreive the
>>> content later.
>>>
>>> Regards
>>> Ganesh
>>>
>>> ----- Original Message ----- From: "Aditi Goyal" <aditigupta20@gmail.com 
>>> >
>>> To: <java-user@lucene.apache.org>
>>> Sent: Friday, October 03, 2008 3:03 PM
>>> Subject: Re: Document larger than setRAMBufferSizeMB()
>>>
>>>
>>> Thanks Anshum.
>>>> Although it raises another query, committing the current buffer  
>>>> will
>>>> commit
>>>> the docs before and what will happen to the current doc which  
>>>> threw an
>>>> error
>>>> while adding a field to it, will that also get committed in the  
>>>> half??
>>>>
>>>> Thanks a lot
>>>> Aditi
>>>>
>>>> On Fri, Oct 3, 2008 at 2:12 PM, Anshum <anshumg@gmail.com> wrote:
>>>>
>>>> Hi Aditi,
>>>>>
>>>>> I guess increasing the buffer size would be a solution here, but  
>>>>> in case
>>>>> you
>>>>> wouldn't know the expected max doc size. I guess the best way to  
>>>>> handle
>>>>> that
>>>>> would be a regular try catch block in which you could commit the  
>>>>> current
>>>>> buffer. At the least you could just continue the loop after doing
>>>>> whatever
>>>>> you wish to do using an exception handling block.
>>>>>
>>>>> --
>>>>> Anshum Gupta
>>>>> Naukri Labs!
>>>>> http://ai-cafe.blogspot.com
>>>>>
>>>>> The facts expressed here belong to everybody, the opinions to  
>>>>> me. The
>>>>> distinction is yours to draw............
>>>>>
>>>>> On Fri, Oct 3, 2008 at 1:56 PM, Aditi Goyal <aditigupta20@gmail.com

>>>>> >
>>>>> wrote:
>>>>>
>>>>>> Hi Everyone,
>>>>>>
>>>>>> I have an index which I am opening at one time only. I keep  
>>>>>> adding the
>>>>>> documents to it until I reach a limit of 500.
>>>>>> After this, I close the index and open it again. (This is done in
>>>>> order
>>>>> to
>>>>>> save time taken by opening and closing the index)
>>>>>> Also, I have set setRAMBufferSizeMB to 16MB.
>>>>>>
>>>>>> If the document size itself is greater than 16MB what will  
>>>>>> happen in >
>>>>> this
>>>>>> case??
>>>>>> It is throwing
>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>> Now, my query is,
>>>>>> Can we change something in the way we parse/index to make it  
>>>>>> more >
>>>>> memory
>>>>>> friendly so that it doesnt throw this exception.
>>>>>> And, Can it be caught and overcome gracefully?
>>>>>>
>>>>>>
>>>>>> Thanks a lot
>>>>>> Aditi
>>>>>>
>>>>>
>>>>>
>>> Send instant messages to your online friends
>>> http://in.messenger.yahoo.com
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message