Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Received-SPF: pass (asf.osuosl.org: domain of yseeley@gmail.com designates
 64.233.166.182 as permitted sender)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
        s=beta; d=gmail.com;
        h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references;
        b=kV/b0xdYhXczwnfhWfM0HyIo5KJo+vGlTjWYrW9EyNgeEgp+GR+csTRwkh9uTNDHLCkkOYis1+KBuTF6jNL0HqhAQSfokEzLl3QKAsvqwZSlQ0R8uKcKIiM8b3vTzh1C6QwUmr8/7HGdyDUEOELYTofxf9h9isbUUetIE8j3NAw=
Message-ID: <c68e39170607131407r7815a466h2259c238adb5ca0a@mail.gmail.com>
Date: Thu, 13 Jul 2006 17:07:03 -0400
From: "Yonik Seeley" <yseeley@gmail.com>
To: java-dev@lucene.apache.org
Subject: Re: [jira] Commented: (LUCENE-565) Supporting deleteDocuments in
 IndexWriter (Code and Performance Results Provided)
In-Reply-To: <c7b302c50607121314t6561f920ocb18a70f568c9a65@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <14645793.1152156330186.JavaMail.jira@brutus>
	 <C7CEFFC0-544E-461D-83AB-F7A70EE02DA2@ix.netcom.com>
	 <c68e39170607060850x69218b97q879d1b092163827a@mail.gmail.com>
	 <20060707130551.GA10773@fermat.math.technion.ac.il>
	 <c68e39170607071043o17327d9ds6015be65e6b800c2@mail.gmail.com>
	 <c7b302c50607092125j43844691q4e91ad3793dd2522@mail.gmail.com>
	 <c68e39170607100927v74d67e53q43c0efe764010dc@mail.gmail.com>
	 <c7b302c50607121228p19955886k91b518c4f563223f@mail.gmail.com>
	 <c68e39170607121252r7142899hd4d7b711f82c1b27@mail.gmail.com>
	 <c7b302c50607121314t6561f920ocb18a70f568c9a65@mail.gmail.com>

On 7/12/06, Ning Li <ning.li.li@gmail.com> wrote:
> > If it can be done in a separate class, using public APIs (or at least
> > with a minimum of protected access), without a loss in performance,
> > then that's the way to go IMO.
>
> This is exactly what I'm asking. Can it be done using public APIs
> without a loss in performance or functionality?

The answer to that is the crux of the matter :-)

Solr's implementation is here:
http://svn.apache.org/viewvc/incubator/solr/trunk/src/java/org/apache/solr/update/DirectUpdateHandler2.java?view=markup
As I said, instead of keeping track of the maxSegment (equiv to the
docid of the in-memory segments), Solr keeps track of the number of
documents not to delete.
So, a delete sets the count to 0, an overwriting add sets the count to
"1", and a non-overwriting add increases the count.

I'll do some speculation on how Solr would compare with NewIndexModifier:
For a very small batch of updates to an existing index:
  - should be little or no difference... they both do the same amount of work.
Building a complete index, without any deletes:
  - no difference
Building a complete index, with deletes
  - there will be some differences...
    Currently, Solr only does the real deletes on a commit call (but
this could be changed).  That means that NewIndexModifier will be
doing deletes more often (every maxBufferedDocs).  The benefit to more
frequent deletes when doing a complete index build is that some of
them will be on a smaller index... deletes very early on in the
process will be faster than those later on when the index is larger.
The downside to more frequent deletes is that more IndexReaders are
opened and closed.
For a large batch of updates (deletes and adds) to an existing index:
  - probably Solr would be faster due to a single delete phase.

For the default Lucene maxBufferedDocs, I would guess Solr's method
would probably be faster in the majority of cases.  As maxBufferedDocs
increased, that advantage would lessen.

-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search server

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org