lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <>
Subject [jira] Commented: (LUCENE-994) Change defaults in IndexWriter to maximize "out of the box" performance
Date Thu, 25 Oct 2007 18:32:50 GMT


Doug Cutting commented on LUCENE-994:

> After adding a bunch of docs and then searching to see if they are there must you pause
for a bit to make sure enough ms have passed?

No.  You could previously never rely on a newly added being visible to search until you called
IndexWriter#close().  Added documents have always been buffered and all buffers were only
flushed by IndexWriter#close().  It used to be the case that the buffer was memory-only and
a fixed number of documents.  So the last up to MaxBufferedDocs added would not yet be visible.

Now there is an IndexWriter#flush() method that flushes buffers without closing the IndexWriter.
 And with the "autocommit=false" feature, nothing is visible to searchers until either #close()
or #flush() is called.  The primary change of concurrent merging is that calls to addDocument()
generally return faster, with merging work done in the background, but concurrent merging
and "autocommit=false" don't fundamentally change the need to call #close() or #flush() in
order to guarantee that all changes are visible to searchers.

At least that's my understanding...

> Change defaults in IndexWriter to maximize "out of the box" performance
> -----------------------------------------------------------------------
>                 Key: LUCENE-994
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 2.3
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 2.3
>         Attachments: LUCENE-994.patch,
> This is follow-through from LUCENE-845, LUCENE-847 and LUCENE-870;
> I'll commit this once those three are committed.
> Out of the box performance of IndexWriter is maximized when flushing
> by RAM instead of a fixed document count (the default today) because
> documents can vary greatly in size.
> Likewise, merging performance should be faster when merging by net
> segment size since, to minimize the net IO cost of merging segments
> over time, you want to merge segments of equal byte size.
> Finally, ConcurrentMergeScheduler improves indexing speed
> substantially (25% in a simple initial test in LUCENE-870) because it
> runs the merges in the backround and doesn't block
> add/update/deleteDocument calls.  Most machines have concurrency
> between CPU and IO and so it makes sense to default to this
> MergeScheduler.
> Note that these changes will break users of ParallelReader because the
> parallel indices will no longer have matching docIDs.  Such users need
> to switch IndexWriter back to flushing by doc count, and switch the
> MergePolicy back to LogDocMergePolicy.  It's likely also necessary to
> switch the MergeScheduler back to SerialMergeScheduler to ensure
> deterministic docID assignment.
> I think the combination of these three default changes, plus other
> performance improvements for indexing (LUCENE-966, LUCENE-843,
> LUCENE-963, LUCENE-969, LUCENE-871, etc.) should make for some sizable
> performance gains Lucene 2.3!

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message