lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jm <jmugur...@gmail.com>
Subject Re: process dies with OOM after processing 10k docs
Date Tue, 16 Dec 2008 10:37:35 GMT
yes I have tested with up to 512MB, althought I dont have the hprof
dump file of those tests, they also got the OOM. I was just wondering
whether having so many instances of FreqProxTermsWriter$PostingList
around is a clear indicator of something I am not releasing or
something.

javier

On Tue, Dec 16, 2008 at 11:24 AM, Ian Lea <ian.lea@gmail.com> wrote:
> Can you not just give the process some more memory?  128Mb seems very
> low for what you are doing.
>
>
> --
> Ian.
>
>
> On Mon, Dec 15, 2008 at 6:28 PM, jm <jmuguruza@gmail.com> wrote:
>> Hi,
>>
>> I am having a memory issue with Lucene2.4. I am strating a process
>> with 128MB of ram, this process handles incoming request from others,
>> and indexes objects in a number of lucene indexes.
>>
>> My lucene docs, all have 6 fields:
>> -one is small: Field.Store.YES, Field.Index.UN_TOKENIZED, Field.TermVector.YES
>> -rest are:  Field.Store.YES, Field.Index.UN_TOKENIZED,
>> Field.TermVector.NO, two of them can be quite large.
>>
>> I set field.setOmitNorms(true); for all fields.
>> And I also set writer.setRAMBufferSizeMB(30);
>>
>> I have archived around 10k lucene docs, they are evenly distritubed in
>> around 100 indexes. I keep a cache of max 60 indexes open, when a new
>> one has to be open for writing, another one is closed.
>> I have started the process with -XX:+HeapDumpOnOutOfMemoryError so I
>> get a hprof file dumped when I get an OOM.
>>
>> I open the hprof file with visualvm and I can see the following:
>> class                                                  instances           size
>> FreqProxTermsWriter$PostingList         572672              20616192
>> String                                                  186154
>>      4467698
>> char[]                                                  184689
>>     30216520
>>
>> In the indexes directories, there are lock files left, as the process
>> dies with the OOM, there are not prx files, but there are frq files (I
>> cannot be sure 100% about that last one but I think so, I dont have
>> access to the server now).
>>
>> I also add each doc into a MemoryIndex, and run a number of
>>  memoryindex.search(lucBooleanQuery);
>> I dont close these MemoryIndex or anything, they just go out of scope,
>> I thought that would be enough to dispose of them.
>>
>> Obviously something in my code is wrong, does someone see anything
>> suspicious here? Any idea where should I be begin looking in my code?
>>
>> thanks a lot
>> javi
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message