Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: <E28B87DDBA121F45B476C053B2171F0E596BDE@mail.intermediate.de>
References: <E28B87DDBA121F45B476C053B2171F0E596BD6@mail.intermediate.de>
	 <9ac0c6aa0906240148ybcd0694ua3b9f7b7788a1c4b@mail.gmail.com>
	 <E28B87DDBA121F45B476C053B2171F0E596BD7@mail.intermediate.de>
	 <9ac0c6aa0906240252v36bb1301u6abdd7e0d5ffbd56@mail.gmail.com>
	 <E28B87DDBA121F45B476C053B2171F0E596BD8@mail.intermediate.de>
	 <9ac0c6aa0906240557l616e9f7ch2b1067398d2f1082@mail.gmail.com>
	 <E28B87DDBA121F45B476C053B2171F0E596BDB@mail.intermediate.de>
	 <9ac0c6aa0906240850o65fe805dp28caa82b0c790da4@mail.gmail.com>
	 <E28B87DDBA121F45B476C053B2171F0E596BDE@mail.intermediate.de>
Date: Thu, 25 Jun 2009 07:13:06 -0400
Message-ID: <9ac0c6aa0906250413sa84469co71587e2bf6b61e24@mail.gmail.com>
Subject: Re: OutOfMemoryError using IndexWriter
From: Michael McCandless <lucene@mikemccandless.com>
To: java-user@lucene.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Can you post your test code?  If you can make it a standalone test,
then I can repro and dig down faster.

Can you try calling IndexWriter.setMaxBufferedDeleteTerms (eg, 1000)
and see if that prevents the OOM?

Mike

On Thu, Jun 25, 2009 at 7:10 AM, stefan<stefan@intermediate.de> wrote:
>
> Hi Mike,
>
> I just changed my test-code to run in an indefinite loop over the databas=
e to index everything. Set the jvm to 120MB heap size, all other parameters=
 as before.
> I got an OOError just as before - so I would say there is a leak somewher=
e.
>
> Here is the histogram.
>
> Heap Histogram
>
> All Classes (excluding platform)
> Class =A0 Instance Count =A0Total Size
> class [B =A0 =A0 =A0 =A01809102 =A0 =A0 =A0 =A0 41992326
> class [C =A0 =A0 =A0 =A0200610 =A026877068
> class [[B =A0 =A0 =A0 46117 =A0 9473872
> class java.lang.String =A0198629 =A03178064
> class org.apache.lucene.index.FreqProxTermsWriter$PostingList =A0 100927 =
=A02825956
> class [Ljava.util.HashMap$Entry; =A0 =A0 =A0 =A011329 =A0 2494312
> class java.util.HashMap$Entry =A0 132578 =A02121248
> class [I =A0 =A0 =A0 =A05186 =A0 =A02097300
>
> So far I had no success in pinpointing those binary arrays, I will need s=
ome more time for this.
>
> Stefan
>
> -----Urspr=FCngliche Nachricht-----
> Von: Michael McCandless [mailto:lucene@mikemccandless.com]
> Gesendet: Mi 24.06.2009 17:50
> An: java-user@lucene.apache.org
> Betreff: Re: OutOfMemoryError using IndexWriter
>
> On Wed, Jun 24, 2009 at 10:18 AM, stefan<stefan@intermediate.de> wrote:
>>
>> Hi,
>>
>>
>>>OK so this means it's not a leak, and instead it's just that stuff is
>>>consuming more RAM than expected.
>> Or that my test db is smaller than the production db which is indeed the=
 case.
>
> But a "leak" would keep leaking over time, right? =A0Ie even a 1 GB heap
> on your test db should eventually throw OOME if there's really a leak.
>
>> Please explain those buffered deletes in a few more details.
>
> Are you calling updateDocument (which deletes then adds)?
>
> Deletes (the Term or Query you pass to updateDocument or
> deleteDocuments) are buffered in a HashMap and then that buffer is
> materialized into actual deleted doc IDs when IndexWriter decides to
> do so. =A0I think IndexWriter isn't properly flushing the deletes when
> they use too much RAM.
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org