lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <>
Subject [jira] Commented: (LUCENE-2755) Some improvements to CMS
Date Tue, 16 Nov 2010 14:38:13 GMT


Michael McCandless commented on LUCENE-2755:

bq. But then you accumulate too many tiny merges, while waiting for the big one to finish?

You say this, as if it was something terribly wrong. 
Big merges aren't heffalumps, they don't usually stalk IW in droves. Big merge ends sooner
or later, and tiny ones go out in a flash.

In fact there is something wrong: without us explicitly scheduling the
running merges (ie setting thread priorities, stopping the big merges
when there are too many small ones), CMS will pause the incoming

I first saw this happen when testing our NRT reopen perf (which is
merge intensive).  Normally the turnaround is very fast (eg 5 msec)
but if you're at your max merge count then CMS will stall you and the
turnaround time easily becomes seconds, which is awful.

It's like an OS that refuses to schedule your "ls" command because
there's still some long running process...

CMS's explicit thread scheduling fixes that problem -- a big merge no
longer causes seconds of delay in opening a new NRT reader.  That is,
as long as net/net you've allocated enough CPUs (maxThreadCount) to do
the merging.  If merging is too slow vs indexing rate + reopen rate
then there's no hope: at some point reopen must be blocked (it's
a zero sum game).

In the ideal world the OS/JRE would do a better job scheduling, ie
realize that there are looong running threads and down prioritize them
(like the OS does to processes), but in practice it's nowhere close to
doing this right for our use case, so I don't see a choice here.  If
we want to keep our fast NRT reopen time, we have to manually schedule
our merge threads.

bq. Maybe we should move BSMP to core and make it the default?

Dunno. The index you end up with is larger than with LogWhateverMP.
But you get a nice benefit of having roughly equal-sized big segments, which is cool for running
collection in parallel.
Everyone has his own requirements.

Fair enough...

bq. But I don't fully understand how it chooses merges. EG does it pick lopsided merges (where
the segments differ substantially in size), as long as they are "small" segments?

Docs say small-sized segments are treated as with LogByteSizeMP.

Hmm... but then how does this differ from setting a maxMergeMB/Docs?

bq. We have seriously inefficient "merge conflict" resolution algorithm on our hands.

You're right!  Though, I think this typically isn't a problem for
LogMP since it doesn't normally pick future merges that conflict with
past ones.

Also the problem is bounded by how long the merge takes to finish.

But I agree we should try to fix this... other MPs could conceivably
do this.

Ugly solution - when MP suggests a merge that is a strict superset of a queued, but not yet
running merge - drop the old one, use the new.
Better solution - instead of asking MP for all the merges it deems reasonable on current index,
we only ask it for "most important" one.
And we do it each time MS has an open slot for execution. This way each merge happening is
the best merge possible at that moment.

This seems dangerous: what's an "important" merge?

How about, instead, we let MP return all eligible merges (like it does
today) but then we replace all previously buffered but not yet running
merges w/ the new merges it returned?  Hmm but this would probably
require giving it access to the buffered-but-not-yet-running merge

bq. About mergeInit, I took a look too and discovered a 140 lines method, so I doubt it does
only "new segment name registration". But perhaps I'm wrong and those are redundant 140 lines

Right -- it's doing lots of stuff.  But the question was whether CMS
really must be calling it itself vs leaving IW.merge to call it as SMS
does.  Ideally only IW.merge should call it (and it becomes private),
which if we take the name assignment out "earlier" seems feasible.

> Some improvements to CMS
> ------------------------
>                 Key: LUCENE-2755
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Shai Erera
>            Assignee: Shai Erera
>            Priority: Minor
>             Fix For: 3.1, 4.0
> While running optimize on a large index, I've noticed several things that got me to read
CMS code more carefully, and find these issues:
> * CMS may hold onto a merge if maxMergeCount is hit. That results in the MergeThreads
taking merges from the IndexWriter until they are exhausted, and only then that blocked merge
will run. I think it's unnecessary that that merge will be blocked.
> * CMS sorts merges by segments size, doc-based and not bytes-based. Since the default
MP is LogByteSizeMP, and I hardly believe people care about doc-based size segments anymore,
I think we should switch the default impl. There are two ways to make it extensible, if we
> ** Have an overridable member/method in CMS that you can extend and override - easy.
> ** Have OneMerge be comparable and let the MP determine the order (e.g. by bytes, docs,
calibrate deletes etc.). Better, but will need to tap into several places in the code, so
more risky and complicated.
> On the go, I'd like to add some documentation to CMS - it's not very easy to read and
> I'll work on a patch.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message