lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Erick Erickson (JIRA)" <>
Subject [jira] [Commented] (LUCENE-7976) Make TieredMergePolicy respect maxSegmentSizeMB and allow singleton merges of very large segments
Date Fri, 04 May 2018 16:50:00 GMT


Erick Erickson commented on LUCENE-7976:


Thanks for taking the time, and yeah, the complexity bothers me too.

I'm reluctant to take the comments out for 8263 as their lifetime is expected to be a few
days after this is checked in and I don't want to have to remember how I got it to work, but
we can if you strongly object.

About tests. There are actually a number of tests for TMP functionality already, in particular
merging down to N segments and forcemerging, what particular parts of the code do you think
should be exercised more?

As for naming/style changes, sure. I always feel like an interloper in the Lucene code;  I'm
perfectly willing to try to keep it consistent with the folks' expectations who, you know
live in the Lucene codebase ;)

As for your other suggestions, I don't know. I've rewritten/refactored/whatever this about
three times already and every time I do it gets complicated again.

bq. I wonder if we can generalize the algorithm here into a single method because in the end
they all do the same thing.

doFindMerges is called from all three operations so I'm a bit puzzled about what this would
look like. Each of the three operations has different initial conditions, eligible segments,
max segment size allowed and the like. Once those are established, they all go through the
same scoring/selection code. Figuring out the initial conditions is "interesting"

There are several areas here that are particularly gnarly and any way to un-gnarl them would
be great.

> the variable number of segments in findForcedMerges. I'd love for there to be exactly
two choices; merge down to one or respect max segment size. Controlling the number of segments
would then be more along the lines of setting maxMergedSegmentsMB in TMP.

> the multi-pass nature of findForcedMerges because it respects the maxMergeAtOnceExplicit.

> Somewhat different decisions need to be made in doFindMerges depending on what type of
merge it is. I'm not a huge fan of passing that enum in. One of the iterations tried to pass
information into an uber-function but that lead to having to pass around segmentsToMerge from
findForcedMerges, which wasn't present in the other two, so passing null in from them was
also ugly.

> I also ran into a couple of issues with findMerges needing to _not_ merge segments if
there weren't enough in the tier, which is exactly the opposite of findForcedMerges, which
itself has conditions around whether it should merge a segment with no deletions or not if
exceeds maxMergedSegmentMB which itself is a variable condition based on whether maxSegments
has been specified.....

Let me know if you're going to take a whack at it, even a skeleton of a different approach
would help me get on the same page.

Meanwhile I can incorporate your other comments, they'll be useful no matter what.

> Make TieredMergePolicy respect maxSegmentSizeMB and allow singleton merges of very large
> -------------------------------------------------------------------------------------------------
>                 Key: LUCENE-7976
>                 URL:
>             Project: Lucene - Core
>          Issue Type: Improvement
>            Reporter: Erick Erickson
>            Assignee: Erick Erickson
>            Priority: Major
>         Attachments: LUCENE-7976.patch, LUCENE-7976.patch, LUCENE-7976.patch, LUCENE-7976.patch,
LUCENE-7976.patch, LUCENE-7976.patch, LUCENE-7976.patch
> We're seeing situations "in the wild" where there are very large indexes (on disk) handled
quite easily in a single Lucene index. This is particularly true as features like docValues
move data into MMapDirectory space. The current TMP algorithm allows on the order of 50% deleted
documents as per a dev list conversation with Mike McCandless (and his blog here:
> Especially in the current era of very large indexes in aggregate, (think many TB) solutions
like "you need to distribute your collection over more shards" become very costly. Additionally,
the tempting "optimize" button exacerbates the issue since once you form, say, a 100G segment
(by optimizing/forceMerging) it is not eligible for merging until 97.5G of the docs in it
are deleted (current default 5G max segment size).
> The proposal here would be to add a new parameter to TMP, something like <maxAllowedPctDeletedInBigSegments>
(no, that's not serious name, suggestions welcome) which would default to 100 (or the same
behavior we have now).
> So if I set this parameter to, say, 20%, and the max segment size stays at 5G, the following
would happen when segments were selected for merging:
> > any segment with > 20% deleted documents would be merged or rewritten NO MATTER
HOW LARGE. There are two cases,
> >> the segment has < 5G "live" docs. In that case it would be merged with smaller
segments to bring the resulting segment up to 5G. If no smaller segments exist, it would just
be rewritten
> >> The segment has > 5G "live" docs (the result of a forceMerge or optimize).
It would be rewritten into a single segment removing all deleted docs no matter how big it
is to start. The 100G example above would be rewritten to an 80G segment for instance.
> Of course this would lead to potentially much more I/O which is why the default would
be the same behavior we see now. As it stands now, though, there's no way to recover from
an optimize/forceMerge except to re-index from scratch. We routinely see 200G-300G Lucene
indexes at this point "in the wild" with 10s of  shards replicated 3 or more times. And that
doesn't even include having these over HDFS.
> Alternatives welcome! Something like the above seems minimally invasive. A new merge
policy is certainly an alternative.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message