lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: OutOfMemoryError using IndexWriter
Date Thu, 25 Jun 2009 10:01:44 GMT
On Thu, Jun 25, 2009 at 3:02 AM, stefan<stefan@intermediate.de> wrote:

>>But a "leak" would keep leaking over time, right?  Ie even a 1 GB heap
>>on your test db should eventually throw OOME if there's really a leak.
> No not necessarily, since I stop indexing ones everything is indexed - I shall try repeated
runs with 120MB.

It indeed looks like IndexWriter doens't account for RAM used by
pending deletes.  I've opened
https://issues.apache.org/jira/browse/LUCENE-1717 for this.  Though
I'd normally expect the amount of extra RAM used to be smallish...

Do you have a high merge factor?  Can you run CheckIndex on your index
(java org.apache.lucene.index.CheckIndex /path/to/index) and post the
output back?

Currently IndexWriter will flush these deletes on kicking off a merge,
or if commit() is called, so one workaround you could try is to call
commit() every so often and see if that improves the RAM usage?

>>Are you calling updateDocument (which deletes then adds)?
> Yes I do, I do not know in my code whether the document is already indexed or not. In
my test case I do delete the
> complete index before the run, so all documents should be new to the index. I still use
update though, since I
> this piece of code is generic.

OK that's a good reason to use updateDocument.

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message