lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <>
Subject [jira] Updated: (LUCENE-845) If you "flush by RAM usage" then IndexWriter may over-merge
Date Wed, 15 Aug 2007 23:59:30 GMT


Michael McCandless updated LUCENE-845:

    Attachment: LUCENE-845.patch

First cut patch.  You have to first apply the most recent patch from

and then apply this patch over it.

This patch has two merge policies:


    This is "backwards compatible" to current merge policy, yet,
    resolve this "over-merge issue" by not using the current setting
    of "maxBufferedDocs" when computing levels.  I think it should
    replace the current LogDocMergePolicy from LUCENE-847.


    Chooses merges according to net size in bytes of all files for a
    segment.  I think we should make this one the default merge
    policy, and also change IndexWriter to flush by RAM by default.

They both subclass from abstract base LogMergePolicy and differ only
in the "size" method which defines how you measure a segment's size (#
docs in that segment or net size in bytes of that segment).

The gist of the approach is the same as the current merge policy: you
generally try to merge segments that are "roughly" the same size
(where size can be doc count or byte size), mergeFactor at a time.

The big difference is instead of starting from maxBufferedDocs and
"going up" to determine level, I start from the max segment size (of
all segments in the index) and "go down" to determine level.  This
resolves the bug because levels are "self-defined" by the segments,
rather than by the current value of maxBufferedDocs on IndexWriter.

I then pick merges exactly the same as the current merge policy: if
any level has >= mergeFactor segments, we merge them.

All tests pass, except:

  * One assert in testAddIndexesNoOptimize which was relying on the
    specific invariants of the current merge policy (it's the same
    assert that LUCENE-847 had changed; this assert is testing
    particular corner cases of the current merge policy).  Changing
    the assertEquals to "4" instead of "3" fixes it.

  * TestLogDocMergePolicy (added in LUCENE-847) doesn't compile
    against the new version above because it's using methods that
    don't exist in the new one.

> If you "flush by RAM usage" then IndexWriter may over-merge
> -----------------------------------------------------------
>                 Key: LUCENE-845
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.1
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>         Attachments: LUCENE-845.patch
> I think a good way to maximize performance of Lucene's indexing for a
> given amount of RAM is to flush (writer.flush()) the added documents
> whenever the RAM usage (writer.ramSizeInBytes()) has crossed the max
> RAM you can afford.
> But, this can confuse the merge policy and cause over-merging, unless
> you set maxBufferedDocs properly.
> This is because the merge policy looks at the current maxBufferedDocs
> to figure out which segments are level 0 (first flushed) or level 1
> (merged from <mergeFactor> level 0 segments).
> I'm not sure how to fix this.  Maybe we can look at net size (bytes)
> of a segment and "infer" level from this?  Still we would have to be
> resilient to the application suddenly increasing the RAM allowed.
> The good news is to workaround this bug I think you just need to
> ensure that your maxBufferedDocs is less than mergeFactor *
> typical-number-of-docs-flushed.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message