Return-Path: Delivered-To: apmail-lucene-java-user-archive@www.apache.org Received: (qmail 38117 invoked from network); 25 Jun 2009 10:02:05 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 25 Jun 2009 10:02:05 -0000 Received: (qmail 85147 invoked by uid 500); 25 Jun 2009 10:02:14 -0000 Delivered-To: apmail-lucene-java-user-archive@lucene.apache.org Received: (qmail 85081 invoked by uid 500); 25 Jun 2009 10:02:14 -0000 Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-user@lucene.apache.org Delivered-To: mailing list java-user@lucene.apache.org Received: (qmail 85071 invoked by uid 99); 25 Jun 2009 10:02:14 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Jun 2009 10:02:14 +0000 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [209.85.210.171] (HELO mail-yx0-f171.google.com) (209.85.210.171) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Jun 2009 10:02:05 +0000 Received: by yxe1 with SMTP id 1so2150416yxe.29 for ; Thu, 25 Jun 2009 03:01:44 -0700 (PDT) MIME-Version: 1.0 Received: by 10.151.132.3 with SMTP id j3mr4331316ybn.96.1245924104205; Thu, 25 Jun 2009 03:01:44 -0700 (PDT) In-Reply-To: References: <9ac0c6aa0906240148ybcd0694ua3b9f7b7788a1c4b@mail.gmail.com> <9ac0c6aa0906240252v36bb1301u6abdd7e0d5ffbd56@mail.gmail.com> <9ac0c6aa0906240557l616e9f7ch2b1067398d2f1082@mail.gmail.com> <9ac0c6aa0906240850o65fe805dp28caa82b0c790da4@mail.gmail.com> Date: Thu, 25 Jun 2009 06:01:44 -0400 Message-ID: <9ac0c6aa0906250301h3de27a64mc12c8ad5fde9d775@mail.gmail.com> Subject: Re: OutOfMemoryError using IndexWriter From: Michael McCandless To: java-user@lucene.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org On Thu, Jun 25, 2009 at 3:02 AM, stefan wrote: >>But a "leak" would keep leaking over time, right? =A0Ie even a 1 GB heap >>on your test db should eventually throw OOME if there's really a leak. > No not necessarily, since I stop indexing ones everything is indexed - I = shall try repeated runs with 120MB. It indeed looks like IndexWriter doens't account for RAM used by pending deletes. I've opened https://issues.apache.org/jira/browse/LUCENE-1717 for this. Though I'd normally expect the amount of extra RAM used to be smallish... Do you have a high merge factor? Can you run CheckIndex on your index (java org.apache.lucene.index.CheckIndex /path/to/index) and post the output back? Currently IndexWriter will flush these deletes on kicking off a merge, or if commit() is called, so one workaround you could try is to call commit() every so often and see if that improves the RAM usage? >>Are you calling updateDocument (which deletes then adds)? > Yes I do, I do not know in my code whether the document is already indexe= d or not. In my test case I do delete the > complete index before the run, so all documents should be new to the inde= x. I still use update though, since I > this piece of code is generic. OK that's a good reason to use updateDocument. Mike --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org For additional commands, e-mail: java-user-help@lucene.apache.org