Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: <49D60D5B.70601@propylon.com>
References: <49D5EF6C.6080805@propylon.com>
	 <359a92830904030514j3a46cdbbvb251bc3ad1d33075@mail.gmail.com>
	 <49D603EA.9000609@propylon.com>
	 <359a92830904030606j736e3f6bx4a2b4e1eed30bdce@mail.gmail.com>
	 <49D60D5B.70601@propylon.com>
Date: Fri, 3 Apr 2009 10:33:32 -0400
Message-ID: <9ac0c6aa0904030733g11c618e3n34ba267fc3b9928e@mail.gmail.com>
Subject: Re: HeapedScorerDoc using all my memory
From: Michael McCandless <lucene@mikemccandless.com>
To: java-user@lucene.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I'm also confused, because ScorerDocQueue should not be used during indexin=
g.

It's used only when scoring boolean "OR" queries.

Are you doing searching in the same JVM as indexing?

Mike

On Fri, Apr 3, 2009 at 9:21 AM, John Byrne <john.byrne@propylon.com> wrote:
> Unfortunately I'm not sure of the exact number. It happened on a machine =
I
> have no access to, and I was just e-mailed a few details of the problem! =
We
> have a JMS queue, where each message is a file to be indexed. There was
> somewhere between 2000 and 10,000 messages processed when it happened.
>
> Because we don't make a distinction between batch and incremental indexin=
g,
> =A0I am closing and re-opening the writer for each document. What you've
> suggested there could well be the problem, that's certainly where I'll lo=
ok
> next.
>
> Thanks for the ideas anyway - I know I really need to come up with some m=
ore
> info on the problem, so I think the next thing I'll do it try to reproduc=
e
> it locally.
>
> -John
>
> Erick Erickson wrote:
>>
>> Hmmmm, that's odd. how many is "a large number of documents"? And
>> what is your index size when things to wonky? (approximately)
>>
>> I can say that other people create very large indexes without this
>> happening,
>> but the only thing that says is that this isn't a *known* problem.
>>
>> Is there any chance you're re-opening the indexwriter without
>> closing the old one (not that you should, you should reuse the writer,
>> but I've put a new in the wrong place by mistake a time or two)....
>>
>> But I confess that I have no idea what happens under the covers when
>> indexing,
>> so I'll have to leave any real insights to the folks who know.
>>
>> Erick
>>
>> On Fri, Apr 3, 2009 at 8:41 AM, John Byrne <john.byrne@propylon.com>
>> wrote:
>>
>>
>>>
>>> The maximum JVM memory is 2GB. Apparently 1.2GB is being used up by thi=
s
>>> class.
>>>
>>> All IndexWriter settings are left as default.
>>>
>>> I haven't tried any changes yet, because the problem so far has on ly
>>> happened in a production environment that I can't mess with. I am
>>> planning
>>> to try reproducing it locally soon, but it takes quite a while before i=
t
>>> happens.
>>>
>>> -John
>>>
>>> Erick Erickson wrote:
>>>
>>>
>>>>
>>>> How much memory are you allocating for the JVM? And what are your
>>>> various indexwriter settings (e.g. MaxBufferedDocs, MaxMergeDocs, etc)=
.
>>>>
>>>> Have you tried different settings in setRamBufferSizeMB?
>>>>
>>>> Best
>>>> Erick
>>>>
>>>> On Fri, Apr 3, 2009 at 7:13 AM, John Byrne <john.byrne@propylon.com>
>>>> wrote:
>>>>
>>>>
>>>>
>>>>
>>>>>
>>>>> Hi, I'm having a problem where the JVM runs out of memory while
>>>>> indexing
>>>>> a
>>>>> large number of files. An analysis of the heapdump shows that most of
>>>>> the
>>>>> memory was taken up with
>>>>> "org/apache/lucene/util/ScorerDocQueue$HeapedScorerDoc".
>>>>>
>>>>> I can't find any leaks in my code so far, and I was wondering, are
>>>>> there
>>>>> any known problems with this class that could leak memory?
>>>>>
>>>>> I'm using version 2.3.1.
>>>>>
>>>>> Thanks!
>>>>>
>>>>> -John
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> =A0-------------------------------------------------------------------=
-----
>>>>
>>>>
>>>> No virus found in this incoming message.
>>>> Checked by AVG - www.avg.com Version: 8.0.238 / Virus Database:
>>>> 270.11.39/2038 - Release Date: 04/02/09 19:07:00
>>>>
>>>>
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>>
>> =A0---------------------------------------------------------------------=
---
>>
>>
>> No virus found in this incoming message.
>> Checked by AVG - www.avg.com Version: 8.0.238 / Virus Database:
>> 270.11.40/2039 - Release Date: 04/03/09 06:19:00
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org