lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Earwin Burrfoot <ear...@gmail.com>
Subject Re: Setting the max number of merge threads across IndexWriters
Date Sun, 01 May 2011 15:19:33 GMT
You don't mean 'static var' under 'global'? I hope, very much.

2011/4/16 Jason Rutherglen <jason.rutherglen@gmail.com>:
> I'd rather not lose [important] functionality.  I think a global max
> thread count is the least intrusive way to go, however I also need to
> see if that's possible.  If so I'll open an issue and post a patch.
>
> 2011/4/15 Shai Erera <serera@gmail.com>:
>> Hi
>> This was raised in LUCENE-2755 (along with other useful refactoring to
>> MS-IW-MP interaction). Here is the relevant comment which addresses Jason's
>> particular
>> issue: https://issues.apache.org/jira/browse/LUCENE-2755?focusedCommentId=12966029&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12966029
>> In short, we can refactor CMS to not hold to an IndexWriter member if we
>> change a lot of the API. But IMO, an ExecutorServiceMS is the right way to
>> go, if you don't mind giving up some CMS features, like controlling thread
>> priority and stalling running threads. In fact, even w/ ExecutorServiceMS
>> you can still achieve some (e.g., stalling), but some juggling will be
>> required.
>> Then, instead of trying to factor out IW members from this MS, you could
>> share the same ES with all MS instances, each will keep a reference to a
>> different IW member. This is just a thought though, I haven't tried it.
>> Shai
>>
>> On Thu, Apr 14, 2011 at 8:23 PM, Earwin Burrfoot <earwin@gmail.com> wrote:
>>>
>>> Can't remember. Probably no. I started an experimental MS api rewrite
>>> (incorporating ability to share MSs between IWs) some time ago, but
>>> never had the time to finish it.
>>>
>>> On Thu, Apr 14, 2011 at 19:56, Simon Willnauer
>>> <simon.willnauer@googlemail.com> wrote:
>>> > On Thu, Apr 14, 2011 at 5:52 PM, Earwin Burrfoot <earwin@gmail.com>
>>> > wrote:
>>> >> I proposed to decouple MergeScheduler from IW (stop keeping a
>>> >> reference to it). Then you can create a single CMS and pass it to all
>>> >> your IWs.
>>> > Yep that was it... is there an issue for this?
>>> >
>>> > simon
>>> >>
>>> >> On Thu, Apr 14, 2011 at 19:40, Jason Rutherglen
>>> >> <jason.rutherglen@gmail.com> wrote:
>>> >>> I think the proposal involved using a ThreadPoolExecutor, which
seemed
>>> >>> to not quite work as well as what we have.  I think it'll be easier
to
>>> >>> simply pass a global context that keeps a counter of the actively
>>> >>> running threads, and pass that into each IW's CMS?
>>> >>>
>>> >>> On Thu, Apr 14, 2011 at 8:25 AM, Simon Willnauer
>>> >>> <simon.willnauer@googlemail.com> wrote:
>>> >>>> On Thu, Apr 14, 2011 at 5:20 PM, Jason Rutherglen
>>> >>>> <jason.rutherglen@gmail.com> wrote:
>>> >>>>> Today the ConcurrentMergeScheduler allows setting the max
thread
>>> >>>>> count and is bound to a single IndexWriter.
>>> >>>>>
>>> >>>>> However in the [common] case of multiple IndexWriters running
in
>>> >>>>> the same process, this disallows one from managing the aggregate
>>> >>>>> number of merge threads executing at any given time.
>>> >>>>>
>>> >>>>> I think this can be fixed, shall I open an issue?
>>> >>>>
>>> >>>> go ahead! I think I have seen this suggestion somewhere maybe
you
>>> >>>> need
>>> >>>> to see if there is one already
>>> >>>>
>>> >>>> simon
>>> >>>>>
>>> >>>>>
>>> >>>>> ---------------------------------------------------------------------
>>> >>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> >>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>> >>>>>
>>> >>>>>
>>> >>>>
>>> >>>> ---------------------------------------------------------------------
>>> >>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> >>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>> >>>>
>>> >>>>
>>> >>>
>>> >>> ---------------------------------------------------------------------
>>> >>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> >>> For additional commands, e-mail: dev-help@lucene.apache.org
>>> >>>
>>> >>>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Kirill Zakharenko/Кирилл Захаренко
>>> >> E-Mail/Jabber: earwin@gmail.com
>>> >> Phone: +7 (495) 683-567-4
>>> >> ICQ: 104465785
>>> >>
>>> >> ---------------------------------------------------------------------
>>> >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> >> For additional commands, e-mail: dev-help@lucene.apache.org
>>> >>
>>> >>
>>> >
>>> > ---------------------------------------------------------------------
>>> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> > For additional commands, e-mail: dev-help@lucene.apache.org
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Kirill Zakharenko/Кирилл Захаренко
>>> E-Mail/Jabber: earwin@gmail.com
>>> Phone: +7 (495) 683-567-4
>>> ICQ: 104465785
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>



-- 
Kirill Zakharenko/Кирилл Захаренко
E-Mail/Jabber: earwin@gmail.com
Phone: +7 (495) 683-567-4
ICQ: 104465785

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message