lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <>
Subject [jira] Commented: (LUCENE-845) If you "flush by RAM usage" then IndexWriter may over-merge
Date Thu, 16 Aug 2007 19:43:31 GMT


Michael McCandless commented on LUCENE-845:

> You may avoid the cost of a bunch of small merges, but then you pay
> the price in searching performance. I'm not sure that's the right
> tradeoff because if someone wanted to optimize for indexing
> performance, they would do more in batches.


It's like we would want to run "partial optimize" (ie, merge the tail
of "small" segments) on demand, only when a reader is about to

Or here's another random idea: maybe IndexReaders should load the tail
of "small segments" into a RAMDirectory, for each one.  Ie, an
IndexReader is given a RAM buffer "budget" and it spends it on any
numerous small segments in the index....?

> How does this work when flushing by MB? If you set
> setRamBufferSizeMB(32), are you guaranteed that you never have more
> than 10 segments less than 32MB (ignoring LEVEL_LOG_SPAN for now) if
> mergeFactor is 10?

No, we have the same challenge of avoiding O(N^2) merge cost.  When
merging by "byte size" of the segments, I don't look at the current
RAM buffer size of the writer.

I feel like there should be a strong separation of "flush params" from
"merge params".

> Almost seems like we need a minSegmentSize parameter too - using
> setRamBufferSizeMB confuses two different but related issues.

Exactly!  I'm thinking that I add "minSegmentSize" to the
LogMergePolicy, which is separate from "maxBufferedDocs" and
"ramBufferSizeMB".  And, we default it to values that seem like an
"acceptable" tradeoff of the cost of O(N^2) merging (based on tests I
will run) vs the cost of slowdown to readers...

I'll run some perf tests.  O(N^2) should be acceptable for certain
segment sizes....

> If you "flush by RAM usage" then IndexWriter may over-merge
> -----------------------------------------------------------
>                 Key: LUCENE-845
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.1
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>         Attachments: LUCENE-845.patch
> I think a good way to maximize performance of Lucene's indexing for a
> given amount of RAM is to flush (writer.flush()) the added documents
> whenever the RAM usage (writer.ramSizeInBytes()) has crossed the max
> RAM you can afford.
> But, this can confuse the merge policy and cause over-merging, unless
> you set maxBufferedDocs properly.
> This is because the merge policy looks at the current maxBufferedDocs
> to figure out which segments are level 0 (first flushed) or level 1
> (merged from <mergeFactor> level 0 segments).
> I'm not sure how to fix this.  Maybe we can look at net size (bytes)
> of a segment and "infer" level from this?  Still we would have to be
> resilient to the application suddenly increasing the RAM allowed.
> The good news is to workaround this bug I think you just need to
> ensure that your maxBufferedDocs is less than mergeFactor *
> typical-number-of-docs-flushed.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message