lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shai Erera (JIRA)" <>
Subject [jira] Commented: (LUCENE-2755) Some improvements to CMS
Date Sun, 14 Nov 2010 09:26:13 GMT


Shai Erera commented on LUCENE-2755:

I've looked into integrating an ExecutorService and I think it can really simplify things,
as long as we can let go of CMS sorting merges by their size. And I think - why should it?
What if we make it to MP's decision? Namely, if you care about which merges run first, have
your MP sort them the way you want, before you return them. If we make OneMerge comparable,
that should be a very trivial extension one has to make to MP (extend MP, override the methods,
call super.method() and then sort accordingly).

If we do that, then SMS and CMS will work the same - execute the merges in the order returned
by MP, only CMS will do so in parallel and SMS will do so synchronously. By reading that you
can already see that later (in a separate issue), we can let go of SMS entirely - it will
be a single-threaded ExecutorService, w/ an implementation to wait until the work competes
(unlike CMS which returns immediately). But that's for another day.

What do you think?

> Some improvements to CMS
> ------------------------
>                 Key: LUCENE-2755
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Shai Erera
>            Assignee: Shai Erera
>            Priority: Minor
>             Fix For: 3.1, 4.0
> While running optimize on a large index, I've noticed several things that got me to read
CMS code more carefully, and find these issues:
> * CMS may hold onto a merge if maxMergeCount is hit. That results in the MergeThreads
taking merges from the IndexWriter until they are exhausted, and only then that blocked merge
will run. I think it's unnecessary that that merge will be blocked.
> * CMS sorts merges by segments size, doc-based and not bytes-based. Since the default
MP is LogByteSizeMP, and I hardly believe people care about doc-based size segments anymore,
I think we should switch the default impl. There are two ways to make it extensible, if we
> ** Have an overridable member/method in CMS that you can extend and override - easy.
> ** Have OneMerge be comparable and let the MP determine the order (e.g. by bytes, docs,
calibrate deletes etc.). Better, but will need to tap into several places in the code, so
more risky and complicated.
> On the go, I'd like to add some documentation to CMS - it's not very easy to read and
> I'll work on a patch.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message