lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stefan" <ste...@intermediate.de>
Subject AW: OutOfMemoryError using IndexWriter
Date Fri, 26 Jun 2009 06:54:44 GMT

Hi Mike,

I ran a burn in test overnight with repeatedly indexing the same db in a loop. I set the heap
size to 120MB and called setMaxBufferedDeleteTerms( 1000), I did not call commit and used
the same index writer.
This test passed  without any errors.

So to wrap this up - I shall call commit every 10000 documents added, the performance impact
on calling setMaxBufferedDeleteTerms is quite significant (I guess somewhere between factor
2-10). I'll check on the speed of commit.
Commit seems to be able to free more memory (since it runs with 100MB heap size).
I'll have to give up on tracking down those binary references, I spent too much time on this
already.

Thanks a lot for your insights.

Stefan




-----Ursprüngliche Nachricht-----
Von: Michael McCandless [mailto:lucene@mikemccandless.com]
Gesendet: Do 25.06.2009 15:57
An: java-user@lucene.apache.org
Betreff: Re: OutOfMemoryError using IndexWriter
 
Interesting that excessive deletes buffering is not your problem...

Even if you can't post the resulting test case, if you can simplify it
& run locally, to rule out anything outside Lucene that's allocating
the byte/char/byte[] arrays, that can help to isolate.

Also, profilers can trace where allocations happened, eg YourKit.

Mike

On Thu, Jun 25, 2009 at 9:08 AM, stefan<stefan@intermediate.de> wrote:
> Hi,
>
> I'm afraid my test setup and code this is far too big.
> What I use lucene for is fairly simple. I have a database with about 150 tables, I iterate
all tables and create for each row a String representation similar to a toString method containing
all database data. This string is then fed together with the primary key to lucene. Full-text
search of my db is then possible. Each document in Lucene represents a row in the database.
>
> I tried calling setMaxBufferedDeleteTerms  with 100MB heap size to no avail, but calling
commit every 10000 documents does help. I assume a commit is similar to creating a new IndexWriter.
>
> HTH,
>
> Stefan
>
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: Michael McCandless [mailto:lucene@mikemccandless.com]
> Gesendet: Do 25.06.2009 13:13
> An: java-user@lucene.apache.org
> Betreff: Re: OutOfMemoryError using IndexWriter
>
> Can you post your test code?  If you can make it a standalone test,
> then I can repro and dig down faster.
>
> Can you try calling IndexWriter.setMaxBufferedDeleteTerms (eg, 1000)
> and see if that prevents the OOM?
>
> Mike
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org




Mime
View raw message