lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shai Erera <ser...@gmail.com>
Subject Re: Setting the max number of merge threads across IndexWriters
Date Sat, 16 Apr 2011 04:18:27 GMT
Hi

This was raised in LUCENE-2755 (along with other useful refactoring to
MS-IW-MP interaction). Here is the relevant comment which addresses Jason's
particular issue:
https://issues.apache.org/jira/browse/LUCENE-2755?focusedCommentId=12966029&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12966029

In short, we can refactor CMS to not hold to an IndexWriter member if we
change a lot of the API. But IMO, an ExecutorServiceMS is the right way to
go, if you don't mind giving up some CMS features, like controlling thread
priority and stalling running threads. In fact, even w/ ExecutorServiceMS
you can still achieve some (e.g., stalling), but some juggling will be
required.

Then, instead of trying to factor out IW members from this MS, you could
share the same ES with all MS instances, each will keep a reference to a
different IW member. This is just a thought though, I haven't tried it.

Shai

On Thu, Apr 14, 2011 at 8:23 PM, Earwin Burrfoot <earwin@gmail.com> wrote:

> Can't remember. Probably no. I started an experimental MS api rewrite
> (incorporating ability to share MSs between IWs) some time ago, but
> never had the time to finish it.
>
> On Thu, Apr 14, 2011 at 19:56, Simon Willnauer
> <simon.willnauer@googlemail.com> wrote:
> > On Thu, Apr 14, 2011 at 5:52 PM, Earwin Burrfoot <earwin@gmail.com>
> wrote:
> >> I proposed to decouple MergeScheduler from IW (stop keeping a
> >> reference to it). Then you can create a single CMS and pass it to all
> >> your IWs.
> > Yep that was it... is there an issue for this?
> >
> > simon
> >>
> >> On Thu, Apr 14, 2011 at 19:40, Jason Rutherglen
> >> <jason.rutherglen@gmail.com> wrote:
> >>> I think the proposal involved using a ThreadPoolExecutor, which seemed
> >>> to not quite work as well as what we have.  I think it'll be easier to
> >>> simply pass a global context that keeps a counter of the actively
> >>> running threads, and pass that into each IW's CMS?
> >>>
> >>> On Thu, Apr 14, 2011 at 8:25 AM, Simon Willnauer
> >>> <simon.willnauer@googlemail.com> wrote:
> >>>> On Thu, Apr 14, 2011 at 5:20 PM, Jason Rutherglen
> >>>> <jason.rutherglen@gmail.com> wrote:
> >>>>> Today the ConcurrentMergeScheduler allows setting the max thread
> >>>>> count and is bound to a single IndexWriter.
> >>>>>
> >>>>> However in the [common] case of multiple IndexWriters running in
> >>>>> the same process, this disallows one from managing the aggregate
> >>>>> number of merge threads executing at any given time.
> >>>>>
> >>>>> I think this can be fixed, shall I open an issue?
> >>>>
> >>>> go ahead! I think I have seen this suggestion somewhere maybe you need
> >>>> to see if there is one already
> >>>>
> >>>> simon
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> >>>>> For additional commands, e-mail: dev-help@lucene.apache.org
> >>>>>
> >>>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> >>>> For additional commands, e-mail: dev-help@lucene.apache.org
> >>>>
> >>>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> >>> For additional commands, e-mail: dev-help@lucene.apache.org
> >>>
> >>>
> >>
> >>
> >>
> >> --
> >> Kirill Zakharenko/Кирилл Захаренко
> >> E-Mail/Jabber: earwin@gmail.com
> >> Phone: +7 (495) 683-567-4
> >> ICQ: 104465785
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: dev-help@lucene.apache.org
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: dev-help@lucene.apache.org
> >
> >
>
>
>
> --
> Kirill Zakharenko/Кирилл Захаренко
> E-Mail/Jabber: earwin@gmail.com
> Phone: +7 (495) 683-567-4
> ICQ: 104465785
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

Mime
View raw message