lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yonik Seeley <ysee...@gmail.com>
Subject Re: [Performance]: IndexWriter again...
Date Mon, 16 May 2005 16:01:57 GMT
I like the idea Paul.

As far as how it should be implemented, perhaps a count of docs in
memory should be kept.  It doesn't seem necessary to traverse all of
the segments on every add (it's a linear operation, and will only
result in a merge every "minMergeDocs" or "maxBufferedDocs").

-Yonik

On 5/16/05, Paul Smith <psmith@aconex.com> wrote:
> In summary, I still firmly believe that the IndexWriter.maybeMergeSegments()
> is chewing a lot more CPU than would be ideal.  So I ran a simple test.  I
> ran the same test I've done before, using mergeFactor(1000)
> maxBufferedDocs(10000), useCompondFile(false), indexing 5 fields (user
> first/lastname/email address)
> 
> As a baseline using the latest SVN source code, I'm getting an indexing rate
> of between 490-515 items/second of a number of runs.
> 
> By applying the attached simple patch to IndexWriter, I'm getting between
> 945-970 of a number of test runs.  That's a significant speed up.  All the
> patch is doing is deferring the call to maybeMergeSegments so it only does
> it every 2000 iterations (2000 is totally arbitrary on my part).
> 
> I've verified with Luke that the index generated contains the same #
> documents, and same # terms, but I have not had a chance to properly setup
> my local environment to run the test cases.  
> 
> Obviously the attached patch is a dirty hack of the highest order. In my
> case I'm re-indexing from scratch every time, so there may be a reason why
> we shouldn't be doing this sort of deferring of method calls.  Perhaps the
> source code is optimized around incremental/batch updates to _existing_
> indexes, but creating a new index, but with a penalty of creating a new
> index performs slower than one would like.
> 
> Perhaps IndexWriter could benefit from another setting that lets one
> configure how often to call maybeMergeSegments()?  That could of course
> confuse more people than it helps.
> 
> I would really appreciate anyones thoughts on this, I'll be very happy to be
> proven wrong because it will just help me understand more of Lucene.  I
> would hope that speeding up indexing would benefit everyone?  Particularly
> the large scale sites out there.
> 
> cheers,
> 
> Paul Smith

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message