lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Rutherglen <jason.rutherg...@gmail.com>
Subject Re: Setting the max number of merge threads across IndexWriters
Date Sat, 16 Apr 2011 04:59:52 GMT
I'd rather not lose [important] functionality.  I think a global max
thread count is the least intrusive way to go, however I also need to
see if that's possible.  If so I'll open an issue and post a patch.

2011/4/15 Shai Erera <serera@gmail.com>:
> Hi
> This was raised in LUCENE-2755 (along with other useful refactoring to
> MS-IW-MP interaction). Here is the relevant comment which addresses Jason's
> particular
> issue: https://issues.apache.org/jira/browse/LUCENE-2755?focusedCommentId=12966029&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12966029
> In short, we can refactor CMS to not hold to an IndexWriter member if we
> change a lot of the API. But IMO, an ExecutorServiceMS is the right way to
> go, if you don't mind giving up some CMS features, like controlling thread
> priority and stalling running threads. In fact, even w/ ExecutorServiceMS
> you can still achieve some (e.g., stalling), but some juggling will be
> required.
> Then, instead of trying to factor out IW members from this MS, you could
> share the same ES with all MS instances, each will keep a reference to a
> different IW member. This is just a thought though, I haven't tried it.
> Shai
>
> On Thu, Apr 14, 2011 at 8:23 PM, Earwin Burrfoot <earwin@gmail.com> wrote:
>>
>> Can't remember. Probably no. I started an experimental MS api rewrite
>> (incorporating ability to share MSs between IWs) some time ago, but
>> never had the time to finish it.
>>
>> On Thu, Apr 14, 2011 at 19:56, Simon Willnauer
>> <simon.willnauer@googlemail.com> wrote:
>> > On Thu, Apr 14, 2011 at 5:52 PM, Earwin Burrfoot <earwin@gmail.com>
>> > wrote:
>> >> I proposed to decouple MergeScheduler from IW (stop keeping a
>> >> reference to it). Then you can create a single CMS and pass it to all
>> >> your IWs.
>> > Yep that was it... is there an issue for this?
>> >
>> > simon
>> >>
>> >> On Thu, Apr 14, 2011 at 19:40, Jason Rutherglen
>> >> <jason.rutherglen@gmail.com> wrote:
>> >>> I think the proposal involved using a ThreadPoolExecutor, which seemed
>> >>> to not quite work as well as what we have.  I think it'll be easier
to
>> >>> simply pass a global context that keeps a counter of the actively
>> >>> running threads, and pass that into each IW's CMS?
>> >>>
>> >>> On Thu, Apr 14, 2011 at 8:25 AM, Simon Willnauer
>> >>> <simon.willnauer@googlemail.com> wrote:
>> >>>> On Thu, Apr 14, 2011 at 5:20 PM, Jason Rutherglen
>> >>>> <jason.rutherglen@gmail.com> wrote:
>> >>>>> Today the ConcurrentMergeScheduler allows setting the max thread
>> >>>>> count and is bound to a single IndexWriter.
>> >>>>>
>> >>>>> However in the [common] case of multiple IndexWriters running
in
>> >>>>> the same process, this disallows one from managing the aggregate
>> >>>>> number of merge threads executing at any given time.
>> >>>>>
>> >>>>> I think this can be fixed, shall I open an issue?
>> >>>>
>> >>>> go ahead! I think I have seen this suggestion somewhere maybe you
>> >>>> need
>> >>>> to see if there is one already
>> >>>>
>> >>>> simon
>> >>>>>
>> >>>>>
>> >>>>> ---------------------------------------------------------------------
>> >>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> >>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>> >>>>>
>> >>>>>
>> >>>>
>> >>>> ---------------------------------------------------------------------
>> >>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> >>>> For additional commands, e-mail: dev-help@lucene.apache.org
>> >>>>
>> >>>>
>> >>>
>> >>> ---------------------------------------------------------------------
>> >>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> >>> For additional commands, e-mail: dev-help@lucene.apache.org
>> >>>
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> Kirill Zakharenko/Кирилл Захаренко
>> >> E-Mail/Jabber: earwin@gmail.com
>> >> Phone: +7 (495) 683-567-4
>> >> ICQ: 104465785
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> >> For additional commands, e-mail: dev-help@lucene.apache.org
>> >>
>> >>
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> > For additional commands, e-mail: dev-help@lucene.apache.org
>> >
>> >
>>
>>
>>
>> --
>> Kirill Zakharenko/Кирилл Захаренко
>> E-Mail/Jabber: earwin@gmail.com
>> Phone: +7 (495) 683-567-4
>> ICQ: 104465785
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message