lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: OutOfMemoryError using IndexWriter
Date Wed, 24 Jun 2009 12:57:38 GMT
On Wed, Jun 24, 2009 at 7:43 AM, stefan<stefan@intermediate.de> wrote:

> I tried with 100MB heap size and got the Error as well, it runs fine with 120MB.

OK so this means it's not a leak, and instead it's just that stuff is
consuming more RAM than expected.

> Here is the histogram (application classes marked with --)
>
> Heap Histogram
>
> All Classes (excluding platform)
> Class   Instance Count  Total Size
> class [C        234200  30245722
> class [B        1087565         25999145
> class [[B       28430   4890060
> class java.lang.String  232351  3717616
> class org.apache.lucene.index.FreqProxTermsWriter$PostingList   99584   2788352
> class java.util.HashMap$Entry   171031  2736496
> class [Ljava.util.HashMap$Entry;        9563    2371256
> class [Ljava.lang.Object;       31820   1820224
> class ---       4474    1753808
> class [I        4337    1567796
> class java.lang.reflect.Method  19774   1364406
> class org.apache.lucene.index.Term      117982  943856
> class [Lorg.apache.lucene.index.RawPostingList;         12      770012
> class ---       1837    490479
> class org.apache.lucene.index.BufferedDeletes$Num       117303  469212
>
> The --- as well was the reflect.Method are part of the app's data.

Hmm -- there are quite a few buffered deletes pending.  It could be we
are under-accounting for RAM used by buffered deletes.  I'll dig on
that.

Also, your char[]'s are taking ~30 MB, byte[] ~26MB, which is odd if
your RAM buffer is 16MB.  Does your app create these?

> Why is it, that creating a new Index Writer will let the indexing run fine with 80MB,
but keeping it will create an
> OutOfMemoryException running with 100MB heap size ?

Could be because you are buffering so many deletes (if indeed Lucene
doesn't account for that RAM consumption properly)...

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message