Return-Path: Delivered-To: apmail-lucene-java-dev-archive@www.apache.org Received: (qmail 96875 invoked from network); 20 Aug 2006 18:36:13 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (209.237.227.199) by minotaur.apache.org with SMTP; 20 Aug 2006 18:36:13 -0000 Received: (qmail 78747 invoked by uid 500); 20 Aug 2006 18:36:09 -0000 Delivered-To: apmail-lucene-java-dev-archive@lucene.apache.org Received: (qmail 78700 invoked by uid 500); 20 Aug 2006 18:36:09 -0000 Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-dev@lucene.apache.org Delivered-To: mailing list java-dev@lucene.apache.org Received: (qmail 78689 invoked by uid 99); 20 Aug 2006 18:36:08 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (140.211.166.49) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 20 Aug 2006 11:36:08 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: neutral (asf.osuosl.org: local policy) Received: from [66.111.4.25] (HELO out1.smtp.messagingengine.com) (66.111.4.25) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 20 Aug 2006 11:35:59 -0700 Received: from frontend3.internal (frontend3.internal [10.202.2.152]) by frontend1.messagingengine.com (Postfix) with ESMTP id 01ADBD9CBB0 for ; Sun, 20 Aug 2006 14:35:32 -0400 (EDT) Received: from heartbeat1.internal ([10.202.2.160]) by frontend3.internal (MEProxy); Sun, 20 Aug 2006 14:35:33 -0400 X-Sasl-enc: VhxCeHKzY4XVVrm5uWBrvgnEDmtJXU/u6otGU5bfff95 1156098932 Received: from [10.17.4.3] (pool-72-70-33-153.bstnma.fios.verizon.net [72.70.33.153]) by mail.messagingengine.com (Postfix) with ESMTP id E898F5FD1 for ; Sun, 20 Aug 2006 14:35:32 -0400 (EDT) Message-ID: <44E8AB73.7000100@mikemccandless.com> Date: Sun, 20 Aug 2006 14:35:31 -0400 From: Michael McCandless User-Agent: Thunderbird 2.0a1 (X11/20060724) MIME-Version: 1.0 To: java-dev@lucene.apache.org Subject: Re: Lock-less commits References: <44E5B16D.4010805@mikemccandless.com> <44E5E364.9010907@mikemccandless.com> <44E609E7.9020406@mikemccandless.com> <44E87EC1.5020707@mikemccandless.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Yonik Seeley wrote: > On 8/20/06, Michael McCandless wrote: >> On deletable: yes, I'm currently GC'ing unused segments by doing a >> full directory listing. > > Actually, you could get a full directory listing once per IndexWriter > and keep the results up-to-date in memory (including deletes that > fail). No need for a "deletable" file, and the directory-listing hit > is only taken once per IndexWriter instance, not once per merge. Excellent! I will take this approach. > IndexWriters also need to open IndexReaders (SegmentReaders) for > merging... I don't know if you needed to modify SegmentReader in a way > that reduces performance, but if so it might be possible to make a > special package protected factory method for use by IndexWriter that > regains any performance loss by making certain assumptions. So far, I believe my mods to SegmentReader should not affect performance. It's just when instantiating the SegmentInfos (well, SegmentInfos.read()) that I do a directory listing to find the latest "generation" segments.N file. When IndexWriter creates the SegmentMerger, since it uses its own SegmentInfo's to get() each SegmentReader, all the necessary details (which .del.N and norms files are "current") are in the SegmentInfo and so SegmentReader doesn't need to do any extra "work". Still this is a good suggestion to remember for future work. Thanks for all the feedback! Mike --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org For additional commands, e-mail: java-dev-help@lucene.apache.org