lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <>
Subject [jira] Created: (LUCENE-856) Optimize segment merging
Date Wed, 04 Apr 2007 21:31:32 GMT
Optimize segment merging

                 Key: LUCENE-856
             Project: Lucene - Java
          Issue Type: Improvement
          Components: Index
    Affects Versions: 2.1
            Reporter: Michael McCandless
         Assigned To: Michael McCandless
            Priority: Minor

With LUCENE-843, the time spent indexing documents has been
substantially reduced and now the time spent merging is a sizable
portion of indexing time.

I ran a test using the patch for LUCENE-843, building an index of 10
million docs, each with ~5,500 byte plain text, with term vectors
(positions + offsets) on and with 2 small stored fields per document.
RAM buffer size was 32 MB.  I didn't optimize the index in the end,
though optimize speed would also improve if we optimize segment
merging.  Index size is 86 GB.

Total time to build the index was 8 hrs 38 minutes, 5 hrs 40 minutes
of which was spent merging.  That's 65.6% of the time!

Most of this time is presumably IO which probably can't be reduced
much unless we improve overall merge policy and experiment with values
for mergeFactor / buffer size.

These tests were run on a Mac Pro with 2 dual-core Intel CPUs.  The IO
system is RAID 0 of 4 drives, so, these times are probably better than
the more common case of a single hard drive which would likely be
slower IO.

I think there are some simple things we could do to speed up merging:

  * Experiment with buffer sizes -- maybe larger buffers for the
    IndexInputs used during merging could help?  Because at a default
    mergeFactor of 10, the disk heads must do alot of seeking back and
    forth between these 10 files (and then to the 11th file where we
    are writing).

  * Use byte copying when possible, eg if there are no deletions on a
    segment we can almost (I think?) just copy things like prox
    postings, stored fields, term vectors, instead of full parsing to
    Jave objects and then re-serializing them.

  * Experiment with mergeFactor / different merge policies.  For
    example I think LUCENE-854 would reduce time spend merging for a
    given index size.

This is currently just a place to list ideas for optimizing segment
merges.  I don't plan on working on this until after LUCENE-843.

Note that for "autoCommit=false", this optimization is somewhat less
important, depending on how often you actually close/open a new
IndexWriter.  In the extreme case, if you open a writer, add 100 MM
docs, close the writer, then no segment merges happen at all.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message