lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stanislav Jordanov <ste...@sirma.bg>
Subject Re: OutOfMemory when indexing
Date Tue, 14 Jun 2005 07:33:14 GMT

Gusenbauer Stefan wrote:

>A few weeks before I had a similar problem too. I will write my problem
>and the solution for it:
>I'm indexing docs and every parsed document is stored in an ArrayList.
>This solution worked for little directories with a little number of
>files in it but when the things are growing you're in trouble.
>My solution was whenever I will run out of memory I will "save" the
>documents. I open the indexwriter and write every document from the
>arraylist to the index. Then I set the arraylist and some other stuff =
>null and try to invoke the garbage collector. Then I do some
>reinitializing and continue indexing.
> Looks easy but it wasn't. How do I check if i will run out of memory?
>Runtimeclass and its methods for getting information about the free
>memory were very unreliable.
>Therefore I changed to Java 1.5 and implemented a memorynotification
>listener which is support by the java.lang.management package. There you
>can adjust a threshold when you should be informed. After the
>notification I perform a "save".
>
>Hope this will help you
>Stefan
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>  
>
Thank you Stefan,
unfortunately, our situation is a bit different - we are not caching 
parsed docs in any way.
When a document is parsed it is indexed immediately.
So in our case it is not the accumulation of documents waiting to be 
indexed that causes the OutOfMemory exception.
I believe it is a "pure lucene" issue - just as at some point when the 
next doc is added to the index, and this causes (perhaps)
the merging of segments, the memory consumption raises drastically.

Regards
Stanislav


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message