lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: HeapedScorerDoc using all my memory
Date Fri, 03 Apr 2009 14:33:32 GMT
I'm also confused, because ScorerDocQueue should not be used during indexing.

It's used only when scoring boolean "OR" queries.

Are you doing searching in the same JVM as indexing?

Mike

On Fri, Apr 3, 2009 at 9:21 AM, John Byrne <john.byrne@propylon.com> wrote:
> Unfortunately I'm not sure of the exact number. It happened on a machine I
> have no access to, and I was just e-mailed a few details of the problem! We
> have a JMS queue, where each message is a file to be indexed. There was
> somewhere between 2000 and 10,000 messages processed when it happened.
>
> Because we don't make a distinction between batch and incremental indexing,
>  I am closing and re-opening the writer for each document. What you've
> suggested there could well be the problem, that's certainly where I'll look
> next.
>
> Thanks for the ideas anyway - I know I really need to come up with some more
> info on the problem, so I think the next thing I'll do it try to reproduce
> it locally.
>
> -John
>
> Erick Erickson wrote:
>>
>> Hmmmm, that's odd. how many is "a large number of documents"? And
>> what is your index size when things to wonky? (approximately)
>>
>> I can say that other people create very large indexes without this
>> happening,
>> but the only thing that says is that this isn't a *known* problem.
>>
>> Is there any chance you're re-opening the indexwriter without
>> closing the old one (not that you should, you should reuse the writer,
>> but I've put a new in the wrong place by mistake a time or two)....
>>
>> But I confess that I have no idea what happens under the covers when
>> indexing,
>> so I'll have to leave any real insights to the folks who know.
>>
>> Erick
>>
>> On Fri, Apr 3, 2009 at 8:41 AM, John Byrne <john.byrne@propylon.com>
>> wrote:
>>
>>
>>>
>>> The maximum JVM memory is 2GB. Apparently 1.2GB is being used up by this
>>> class.
>>>
>>> All IndexWriter settings are left as default.
>>>
>>> I haven't tried any changes yet, because the problem so far has on ly
>>> happened in a production environment that I can't mess with. I am
>>> planning
>>> to try reproducing it locally soon, but it takes quite a while before it
>>> happens.
>>>
>>> -John
>>>
>>> Erick Erickson wrote:
>>>
>>>
>>>>
>>>> How much memory are you allocating for the JVM? And what are your
>>>> various indexwriter settings (e.g. MaxBufferedDocs, MaxMergeDocs, etc).
>>>>
>>>> Have you tried different settings in setRamBufferSizeMB?
>>>>
>>>> Best
>>>> Erick
>>>>
>>>> On Fri, Apr 3, 2009 at 7:13 AM, John Byrne <john.byrne@propylon.com>
>>>> wrote:
>>>>
>>>>
>>>>
>>>>
>>>>>
>>>>> Hi, I'm having a problem where the JVM runs out of memory while
>>>>> indexing
>>>>> a
>>>>> large number of files. An analysis of the heapdump shows that most of
>>>>> the
>>>>> memory was taken up with
>>>>> "org/apache/lucene/util/ScorerDocQueue$HeapedScorerDoc".
>>>>>
>>>>> I can't find any leaks in my code so far, and I was wondering, are
>>>>> there
>>>>> any known problems with this class that could leak memory?
>>>>>
>>>>> I'm using version 2.3.1.
>>>>>
>>>>> Thanks!
>>>>>
>>>>> -John
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>  ------------------------------------------------------------------------
>>>>
>>>>
>>>> No virus found in this incoming message.
>>>> Checked by AVG - www.avg.com Version: 8.0.238 / Virus Database:
>>>> 270.11.39/2038 - Release Date: 04/02/09 19:07:00
>>>>
>>>>
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>>
>>  ------------------------------------------------------------------------
>>
>>
>> No virus found in this incoming message.
>> Checked by AVG - www.avg.com Version: 8.0.238 / Virus Database:
>> 270.11.40/2039 - Release Date: 04/03/09 06:19:00
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message