lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Upayavira ...@odoko.co.uk>
Subject Re: optimize status
Date Mon, 29 Jun 2015 20:29:27 GMT
For the sake of history, somewhere around Solr/Lucene 3.2 a new
"MergePolicy" was introduced. The old one merged simply based upon age,
or "index generation", meaning the older the segment, the less likely it
would get merged, hence needing optimize to clear out deletes from your
older segments.

The new MergePolicy, the TieredMergePolicy, uses a more intelligent
algorithm to decide which segments to merge, and is the single reason
why optimization isn't recommended anymore. According to the javadocs:

"For normal merging, this policy first computes a "budget" of how many
segments are allowed to be in the index. If the index is over-budget,
then the policy sorts segments by decreasing size (pro-rating by percent
deletes), and then finds the least-cost merge. Merge cost is measured by
a combination of the "skew" of the merge (size of largest segment
divided by smallest segment), total merge size and percent deletes
reclaimed, so that merges with lower skew, smaller size and those
reclaiming more deletes, are favored.

If a merge will produce a segment that's larger than
setMaxMergedSegmentMB(double), then the policy will merge fewer segments
(down to 1 at once, if that one has deletions) to keep the segment size
under budget."

Upayavira


On Mon, Jun 29, 2015, at 08:55 PM, Toke Eskildsen wrote:
> Reitzel, Charles <Charles.Reitzel@tiaa-cref.org> wrote:
> > Is there really a good reason to consolidate down to a single segment?
> 
> In the  scenario spawning this thread it does not seem to be the best
> choice. Speaking more broadly there are Solr setups out there that deals
> with immutable data, often tied to a point in time, e.g. log data. We
> have such a setup (harvested web resources) and are able to lower heap
> requirements significantly and increase speed by building fully optimized
> and immutable shards.
> 
> > Any incremental query performance benefit is tiny compared to the loss of managability.
> 
> True in many cases and I agree that the "Optimize"-wording is a bit of a
> trap. While technically correct, it implies that one should do it
> occasionally to keep any index fit. A different wording and maybe a
> tooltip saying something like "Only recommended for non-changing indexes"
> might be better.
> 
> Turning it around: To minimize the risk of occasional
> performance-degrading large merges, one might want an index where all the
> shards are below a certain size. Splitting larger shards into smaller
> ones would in that case also be an optimization, just towards a different
> goal.
> 
> - Toke Eskildsen

Mime
View raw message