lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jm <jmugur...@gmail.com>
Subject Re: process dies with OOM after processing 10k docs
Date Tue, 16 Dec 2008 11:21:29 GMT
right...I was forgetting the 30MB flush by ram is PER writer....I'll
make some tests to verify this and fix accordingly...

Thanks!!

On Tue, Dec 16, 2008 at 12:06 PM, Michael McCandless
<lucene@mikemccandless.com> wrote:
>
> That class is what's used to buffer the added docs in IndexWriter.  The heap
> dump seems to indicate you've got ~55 MB worth of buffered docs pending.
>  Since you allow a 30MB RAM buffer for each writer, and it seems like you
> allow up to 60 writers to be opened at once, it seems like in the worst case
> you need to allow for at least 60*30 = 1800 MB RAM consumption just for the
> writers?
>
> Mike
>
> jm wrote:
>
>> yes I have tested with up to 512MB, althought I dont have the hprof
>> dump file of those tests, they also got the OOM. I was just wondering
>> whether having so many instances of FreqProxTermsWriter$PostingList
>> around is a clear indicator of something I am not releasing or
>> something.
>>
>> javier
>>
>> On Tue, Dec 16, 2008 at 11:24 AM, Ian Lea <ian.lea@gmail.com> wrote:
>>>
>>> Can you not just give the process some more memory?  128Mb seems very
>>> low for what you are doing.
>>>
>>>
>>> --
>>> Ian.
>>>
>>>
>>> On Mon, Dec 15, 2008 at 6:28 PM, jm <jmuguruza@gmail.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I am having a memory issue with Lucene2.4. I am strating a process
>>>> with 128MB of ram, this process handles incoming request from others,
>>>> and indexes objects in a number of lucene indexes.
>>>>
>>>> My lucene docs, all have 6 fields:
>>>> -one is small: Field.Store.YES, Field.Index.UN_TOKENIZED,
>>>> Field.TermVector.YES
>>>> -rest are:  Field.Store.YES, Field.Index.UN_TOKENIZED,
>>>> Field.TermVector.NO, two of them can be quite large.
>>>>
>>>> I set field.setOmitNorms(true); for all fields.
>>>> And I also set writer.setRAMBufferSizeMB(30);
>>>>
>>>> I have archived around 10k lucene docs, they are evenly distritubed in
>>>> around 100 indexes. I keep a cache of max 60 indexes open, when a new
>>>> one has to be open for writing, another one is closed.
>>>> I have started the process with -XX:+HeapDumpOnOutOfMemoryError so I
>>>> get a hprof file dumped when I get an OOM.
>>>>
>>>> I open the hprof file with visualvm and I can see the following:
>>>> class                                                  instances
>>>>   size
>>>> FreqProxTermsWriter$PostingList         572672              20616192
>>>> String                                                  186154
>>>>    4467698
>>>> char[]                                                  184689
>>>>   30216520
>>>>
>>>> In the indexes directories, there are lock files left, as the process
>>>> dies with the OOM, there are not prx files, but there are frq files (I
>>>> cannot be sure 100% about that last one but I think so, I dont have
>>>> access to the server now).
>>>>
>>>> I also add each doc into a MemoryIndex, and run a number of
>>>> memoryindex.search(lucBooleanQuery);
>>>> I dont close these MemoryIndex or anything, they just go out of scope,
>>>> I thought that would be enough to dispose of them.
>>>>
>>>> Obviously something in my code is wrong, does someone see anything
>>>> suspicious here? Any idea where should I be begin looking in my code?
>>>>
>>>> thanks a lot
>>>> javi
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message