lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ian Lea" <ian....@gmail.com>
Subject Re: process dies with OOM after processing 10k docs
Date Tue, 16 Dec 2008 10:24:28 GMT
Can you not just give the process some more memory?  128Mb seems very
low for what you are doing.


--
Ian.


On Mon, Dec 15, 2008 at 6:28 PM, jm <jmuguruza@gmail.com> wrote:
> Hi,
>
> I am having a memory issue with Lucene2.4. I am strating a process
> with 128MB of ram, this process handles incoming request from others,
> and indexes objects in a number of lucene indexes.
>
> My lucene docs, all have 6 fields:
> -one is small: Field.Store.YES, Field.Index.UN_TOKENIZED, Field.TermVector.YES
> -rest are:  Field.Store.YES, Field.Index.UN_TOKENIZED,
> Field.TermVector.NO, two of them can be quite large.
>
> I set field.setOmitNorms(true); for all fields.
> And I also set writer.setRAMBufferSizeMB(30);
>
> I have archived around 10k lucene docs, they are evenly distritubed in
> around 100 indexes. I keep a cache of max 60 indexes open, when a new
> one has to be open for writing, another one is closed.
> I have started the process with -XX:+HeapDumpOnOutOfMemoryError so I
> get a hprof file dumped when I get an OOM.
>
> I open the hprof file with visualvm and I can see the following:
> class                                                  instances           size
> FreqProxTermsWriter$PostingList         572672              20616192
> String                                                  186154
>      4467698
> char[]                                                  184689
>     30216520
>
> In the indexes directories, there are lock files left, as the process
> dies with the OOM, there are not prx files, but there are frq files (I
> cannot be sure 100% about that last one but I think so, I dont have
> access to the server now).
>
> I also add each doc into a MemoryIndex, and run a number of
>  memoryindex.search(lucBooleanQuery);
> I dont close these MemoryIndex or anything, they just go out of scope,
> I thought that would be enough to dispose of them.
>
> Obviously something in my code is wrong, does someone see anything
> suspicious here? Any idea where should I be begin looking in my code?
>
> thanks a lot
> javi
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message