lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jm <>
Subject process dies with OOM after processing 10k docs
Date Mon, 15 Dec 2008 18:28:57 GMT

I am having a memory issue with Lucene2.4. I am strating a process
with 128MB of ram, this process handles incoming request from others,
and indexes objects in a number of lucene indexes.

My lucene docs, all have 6 fields:
-one is small: Field.Store.YES, Field.Index.UN_TOKENIZED, Field.TermVector.YES
-rest are:  Field.Store.YES, Field.Index.UN_TOKENIZED,
Field.TermVector.NO, two of them can be quite large.

I set field.setOmitNorms(true); for all fields.
And I also set writer.setRAMBufferSizeMB(30);

I have archived around 10k lucene docs, they are evenly distritubed in
around 100 indexes. I keep a cache of max 60 indexes open, when a new
one has to be open for writing, another one is closed.
I have started the process with -XX:+HeapDumpOnOutOfMemoryError so I
get a hprof file dumped when I get an OOM.

I open the hprof file with visualvm and I can see the following:
class                                                  instances           size
FreqProxTermsWriter$PostingList         572672              20616192
String                                                  186154
char[]                                                  184689

In the indexes directories, there are lock files left, as the process
dies with the OOM, there are not prx files, but there are frq files (I
cannot be sure 100% about that last one but I think so, I dont have
access to the server now).

I also add each doc into a MemoryIndex, and run a number of;
I dont close these MemoryIndex or anything, they just go out of scope,
I thought that would be enough to dispose of them.

Obviously something in my code is wrong, does someone see anything
suspicious here? Any idea where should I be begin looking in my code?

thanks a lot

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message