lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Earwin Burrfoot <ear...@gmail.com>
Subject Re: Thoughts on CMS and SMS
Date Fri, 28 May 2010 14:45:38 GMT
We just need an Executor-based MS, then we can throw all other out of
the window, as threading concerns are now resolved by a proper choice
of Executor supplied to constructor.
Also an application has much more control over threading in
multiple-index situations, as single Executor can be reused for
multiple MSs.

On Fri, May 28, 2010 at 18:36, Michael McCandless
<lucene@mikemccandless.com> wrote:
> Hmm... so I think the questions really are "how many merges are
> allowed to run concurrently?" (SMS is 1 and CMS is N), and "do I spawn
> my own threads for merging or do I steal the app's threads" (SMS
> steals app threads and CMS spawns new ones).
>
> Of course if you steal app threads you can only make use of as much
> concurrency as the app's threads...
>
> Both SMS and CMS will allow other indexing ops to proceed, to a point,
> but if the other indexing ops spawn too many merges, then those
> threads will be blocked by both SMS and CMS.
>
> So I'm not sure blocking/non-blocking is a good first split -- even
> SMS isn't blocking other app indexing threads.
>
> Mike
>
> On Thu, May 27, 2010 at 3:58 AM, Shai Erera <serera@gmail.com> wrote:
>> Hi
>>
>> I've been thinking recently why are these two named like they are ... with a
>> MS we're basically asking two questions: (1) should it block other merges
>> from happening (or app thread from continuing) and (2) should it do its
>> merges concurrently?
>>
>> SMS answers 'true' to (1) and 'false' to (2), while CMS answers the
>> opposite.
>>
>> BUT, there's really no reason why these two are coupled. E.g. someone who
>> wants to block other merges from running, or hold the app thread until
>> merges are finished, does not necessarily want the merges to run in
>> sequence. Those are two different decisions. Even if you want to block the
>> application thread and other merges, you can still benefit form having the
>> merges run concurrently.
>>
>> So, I was thinking that what we really want is a BlockingMS and
>> NonBlockingMS that can be used according to the answer you look for in (1),
>> and then we can have variants for both that execute the merges concurrently
>> or not. I think that serial merging should be supported w/ BlockingMS only,
>> but am interested in other opinions. One of the scenarios for serial merging
>> is if the application wants to ensure no additional threads are spawned
>> other than what it decided to spawn, and therefore it can only be used w/
>> the BlockingMS.Another scenario is to control IO, but in this case a
>> NonBlockingSerialMS may fit as well (depends if you think other merges may
>> start while this one is running).
>>
>> In fact, w/o changing much, we can have CMS optionally block other merges /
>> app thread by waiting until all merges are finished. We may even stick w/
>> the current SMS/CMS names, just documenting that CMS can be used to block
>> threads, only merges will be done concurrently.
>>
>> What do you think?
>>
>> Shai
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>



-- 
Kirill Zakharenko/Кирилл Захаренко (earwin@gmail.com)
Phone: +7 (495) 683-567-4
ICQ: 104465785

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message