Return-Path: Delivered-To: apmail-lucene-java-dev-archive@www.apache.org Received: (qmail 69077 invoked from network); 17 Mar 2010 19:30:54 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 17 Mar 2010 19:30:54 -0000 Received: (qmail 92773 invoked by uid 500); 17 Mar 2010 19:30:53 -0000 Delivered-To: apmail-lucene-java-dev-archive@lucene.apache.org Received: (qmail 92729 invoked by uid 500); 17 Mar 2010 19:30:53 -0000 Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-dev@lucene.apache.org Delivered-To: mailing list java-dev@lucene.apache.org Received: (qmail 92721 invoked by uid 99); 17 Mar 2010 19:30:53 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Mar 2010 19:30:53 +0000 X-ASF-Spam-Status: No, hits=0.7 required=10.0 tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Mar 2010 19:30:46 +0000 Received: by gwj17 with SMTP id 17so586785gwj.35 for ; Wed, 17 Mar 2010 12:30:25 -0700 (PDT) MIME-Version: 1.0 Received: by 10.150.59.21 with SMTP id h21mr1110849yba.3.1268854225400; Wed, 17 Mar 2010 12:30:25 -0700 (PDT) In-Reply-To: <20100317181547.90050@gmx.net> References: <20100317181547.90050@gmx.net> Date: Wed, 17 Mar 2010 14:30:25 -0500 Message-ID: <9ac0c6aa1003171230k18f74894i1d60f91704e1e62d@mail.gmail.com> Subject: Re: IndexWriter.synced field accumulates data From: Michael McCandless To: java-dev@lucene.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org You're right! Really we should delete from sync'd when we delete the files. We need to tie into IndexFileDeleter for that, maybe moving this set into there. Though in practice the amount of actual RAM used should rarely be an issue? But we should fix it... Can you open an issue? Mike On Wed, Mar 17, 2010 at 1:15 PM, Gregor Kaczor wrote: > I am running into a strange OutOfMemoryError. My small test application d= oes index and delete some few files. This is repeated for 60k times. =A0Opt= imization is run from every 2k times a file is indexed. Index size is 50KB.= I did analyze the HeapDumpFile and realized that IndexWriter.synced field = occupied more than half of the heap. That field is a private HashSet withou= t a getter. Its task is to hold files which have been synced already. > > There are two calls to addAll and one call to add on synced but no remove= or clear throughout the lifecycle of the IndexWriter instance. > > According to the Eclipse Memory Analyzer synced contains 32618 entries wh= ich look like file names "_e065_1.del" or "_e067.cfs" > > The index directory contains 10 files only. > > I guess synced is holding obsolete data > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org > For additional commands, e-mail: java-dev-help@lucene.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org For additional commands, e-mail: java-dev-help@lucene.apache.org