lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Byrne <john.by...@propylon.com>
Subject Re: HeapedScorerDoc using all my memory
Date Fri, 03 Apr 2009 13:21:31 GMT
Unfortunately I'm not sure of the exact number. It happened on a machine 
I have no access to, and I was just e-mailed a few details of the 
problem! We have a JMS queue, where each message is a file to be 
indexed. There was somewhere between 2000 and 10,000 messages processed 
when it happened.

Because we don't make a distinction between batch and incremental 
indexing,  I am closing and re-opening the writer for each document. 
What you've suggested there could well be the problem, that's certainly 
where I'll look next.

Thanks for the ideas anyway - I know I really need to come up with some 
more info on the problem, so I think the next thing I'll do it try to 
reproduce it locally.

-John

Erick Erickson wrote:
> Hmmmm, that's odd. how many is "a large number of documents"? And
> what is your index size when things to wonky? (approximately)
>
> I can say that other people create very large indexes without this
> happening,
> but the only thing that says is that this isn't a *known* problem.
>
> Is there any chance you're re-opening the indexwriter without
> closing the old one (not that you should, you should reuse the writer,
> but I've put a new in the wrong place by mistake a time or two)....
>
> But I confess that I have no idea what happens under the covers when
> indexing,
> so I'll have to leave any real insights to the folks who know.
>
> Erick
>
> On Fri, Apr 3, 2009 at 8:41 AM, John Byrne <john.byrne@propylon.com> wrote:
>
>   
>> The maximum JVM memory is 2GB. Apparently 1.2GB is being used up by this
>> class.
>>
>> All IndexWriter settings are left as default.
>>
>> I haven't tried any changes yet, because the problem so far has on ly
>> happened in a production environment that I can't mess with. I am planning
>> to try reproducing it locally soon, but it takes quite a while before it
>> happens.
>>
>> -John
>>
>> Erick Erickson wrote:
>>
>>     
>>> How much memory are you allocating for the JVM? And what are your
>>> various indexwriter settings (e.g. MaxBufferedDocs, MaxMergeDocs, etc).
>>>
>>> Have you tried different settings in setRamBufferSizeMB?
>>>
>>> Best
>>> Erick
>>>
>>> On Fri, Apr 3, 2009 at 7:13 AM, John Byrne <john.byrne@propylon.com>
>>> wrote:
>>>
>>>
>>>
>>>       
>>>> Hi, I'm having a problem where the JVM runs out of memory while indexing
>>>> a
>>>> large number of files. An analysis of the heapdump shows that most of the
>>>> memory was taken up with
>>>> "org/apache/lucene/util/ScorerDocQueue$HeapedScorerDoc".
>>>>
>>>> I can't find any leaks in my code so far, and I was wondering, are there
>>>> any known problems with this class that could leak memory?
>>>>
>>>> I'm using version 2.3.1.
>>>>
>>>> Thanks!
>>>>
>>>> -John
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>>>
>>>>         
>>>  ------------------------------------------------------------------------
>>>
>>>
>>> No virus found in this incoming message.
>>> Checked by AVG - www.avg.com Version: 8.0.238 / Virus Database:
>>> 270.11.39/2038 - Release Date: 04/02/09 19:07:00
>>>
>>>
>>>
>>>       
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>     
>
>   
> ------------------------------------------------------------------------
>
>
> No virus found in this incoming message.
> Checked by AVG - www.avg.com 
> Version: 8.0.238 / Virus Database: 270.11.40/2039 - Release Date: 04/03/09 06:19:00
>
>   


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message