lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Document larger than setRAMBufferSizeMB()
Date Fri, 03 Oct 2008 11:27:35 GMT

Note that large stored fields do not use up any RAM in IndexWriter's  
RAM buffer because these stored fields are immediately written to the  
directory and not stored in RAM for very long.

Aditi, I'd love to see the full stack trace of the OOME that was  
originally hit if you still have it...

Mike

Ganesh wrote:

> Single document of 16 MB seems to be big. I think you are trying to  
> store the entire document content. If it is so drop the stored field  
> and store its reference information in the database, which could  
> help to retreive the content later.
>
> Regards
> Ganesh
>
> ----- Original Message ----- From: "Aditi Goyal" <aditigupta20@gmail.com 
> >
> To: <java-user@lucene.apache.org>
> Sent: Friday, October 03, 2008 3:03 PM
> Subject: Re: Document larger than setRAMBufferSizeMB()
>
>
>> Thanks Anshum.
>> Although it raises another query, committing the current buffer  
>> will commit
>> the docs before and what will happen to the current doc which threw  
>> an error
>> while adding a field to it, will that also get committed in the  
>> half??
>>
>> Thanks a lot
>> Aditi
>>
>> On Fri, Oct 3, 2008 at 2:12 PM, Anshum <anshumg@gmail.com> wrote:
>>
>>> Hi Aditi,
>>>
>>> I guess increasing the buffer size would be a solution here, but  
>>> in case
>>> you
>>> wouldn't know the expected max doc size. I guess the best way to  
>>> handle
>>> that
>>> would be a regular try catch block in which you could commit the  
>>> current
>>> buffer. At the least you could just continue the loop after doing  
>>> whatever
>>> you wish to do using an exception handling block.
>>>
>>> --
>>> Anshum Gupta
>>> Naukri Labs!
>>> http://ai-cafe.blogspot.com
>>>
>>> The facts expressed here belong to everybody, the opinions to me.  
>>> The
>>> distinction is yours to draw............
>>>
>>> On Fri, Oct 3, 2008 at 1:56 PM, Aditi Goyal <aditigupta20@gmail.com>
>>> wrote:
>>>
>>> > Hi Everyone,
>>> >
>>> > I have an index which I am opening at one time only. I keep  
>>> adding the
>>> > documents to it until I reach a limit of 500.
>>> > After this, I close the index and open it again. (This is done  
>>> in order
>>> to
>>> > save time taken by opening and closing the index)
>>> > Also, I have set setRAMBufferSizeMB to 16MB.
>>> >
>>> > If the document size itself is greater than 16MB what will  
>>> happen in > this
>>> > case??
>>> > It is throwing
>>> > java.lang.OutOfMemoryError: Java heap space
>>> > Now, my query is,
>>> > Can we change something in the way we parse/index to make it  
>>> more > memory
>>> > friendly so that it doesnt throw this exception.
>>> > And, Can it be caught and overcome gracefully?
>>> >
>>> >
>>> > Thanks a lot
>>> > Aditi
>>> >
>>>
>
> Send instant messages to your online friends http://in.messenger.yahoo.com
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message