From dev-return-69234-apmail-lucene-dev-archive=lucene.apache.org@lucene.apache.org Sun May 1 15:20:02 2011 Return-Path: X-Original-To: apmail-lucene-dev-archive@www.apache.org Delivered-To: apmail-lucene-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B975E2D03 for ; Sun, 1 May 2011 15:20:02 +0000 (UTC) Received: (qmail 72032 invoked by uid 500); 1 May 2011 15:20:01 -0000 Delivered-To: apmail-lucene-dev-archive@lucene.apache.org Received: (qmail 71980 invoked by uid 500); 1 May 2011 15:20:01 -0000 Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@lucene.apache.org Delivered-To: mailing list dev@lucene.apache.org Received: (qmail 71973 invoked by uid 99); 1 May 2011 15:20:01 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 May 2011 15:20:01 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of earwin@gmail.com designates 209.85.216.169 as permitted sender) Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 May 2011 15:19:55 +0000 Received: by qyk2 with SMTP id 2so1157524qyk.14 for ; Sun, 01 May 2011 08:19:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type:content-transfer-encoding; bh=ifGQTtfbf1Z9m/fgnFWc/8FsvaftkGzEXfPCSOwZVRI=; b=wAadLRbszvB0356AFWQA162+EEp9/zkUUWoNEcaTHYFLGgH0MwBwe9cKrYQww0/uaG L8HuKqxNLgOfbtkOiNp77RzDQf5Lcm8VHDkcnKnwxU8er73bkcVeOJLPfSd1g70dyDMy GARTmUXjR0YRVVdcSLiDt3OVhSZMXRLdxLUNo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=i3Si5E75KQfb6NU06a5Dp2wyBEWtjCSDmjJmwwuvyfVQejE5lbRMl/+tlv5OudwG9o qkOFA4oolrH47olXp5Gl9VHJlNpKh4/6yosdWJRJLAuywCl6JQFZozrBjmZHP3cr7olZ f+DHo9tFn2HX00P1qF7N/XbI0EEf3kO0t7Vao= MIME-Version: 1.0 Received: by 10.229.136.208 with SMTP id s16mr5473347qct.134.1304263173993; Sun, 01 May 2011 08:19:33 -0700 (PDT) Received: by 10.229.29.74 with HTTP; Sun, 1 May 2011 08:19:33 -0700 (PDT) In-Reply-To: References: Date: Sun, 1 May 2011 19:19:33 +0400 Message-ID: Subject: Re: Setting the max number of merge threads across IndexWriters From: Earwin Burrfoot To: dev@lucene.apache.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org You don't mean 'static var' under 'global'? I hope, very much. 2011/4/16 Jason Rutherglen : > I'd rather not lose [important] functionality. =C2=A0I think a global max > thread count is the least intrusive way to go, however I also need to > see if that's possible. =C2=A0If so I'll open an issue and post a patch. > > 2011/4/15 Shai Erera : >> Hi >> This was raised in LUCENE-2755 (along with other useful refactoring to >> MS-IW-MP interaction). Here is the relevant comment which addresses Jaso= n's >> particular >> issue:=C2=A0https://issues.apache.org/jira/browse/LUCENE-2755?focusedCom= mentId=3D12966029&page=3Dcom.atlassian.jira.plugin.system.issuetabpanels:co= mment-tabpanel#comment-12966029 >> In short, we can refactor CMS to not hold to an IndexWriter member if we >> change a lot of the API. But IMO, an ExecutorServiceMS is the right way = to >> go, if you don't mind giving up some CMS features, like controlling thre= ad >> priority and stalling running threads. In fact, even w/ ExecutorServiceM= S >> you can still achieve some (e.g., stalling), but some juggling will be >> required. >> Then, instead of trying to factor out IW members from this MS, you could >> share the same ES with all MS instances, each will keep a reference to a >> different IW member. This is just a thought though, I haven't tried it. >> Shai >> >> On Thu, Apr 14, 2011 at 8:23 PM, Earwin Burrfoot wrot= e: >>> >>> Can't remember. Probably no. I started an experimental MS api rewrite >>> (incorporating ability to share MSs between IWs) some time ago, but >>> never had the time to finish it. >>> >>> On Thu, Apr 14, 2011 at 19:56, Simon Willnauer >>> wrote: >>> > On Thu, Apr 14, 2011 at 5:52 PM, Earwin Burrfoot >>> > wrote: >>> >> I proposed to decouple MergeScheduler from IW (stop keeping a >>> >> reference to it). Then you can create a single CMS and pass it to al= l >>> >> your IWs. >>> > Yep that was it... is there an issue for this? >>> > >>> > simon >>> >> >>> >> On Thu, Apr 14, 2011 at 19:40, Jason Rutherglen >>> >> wrote: >>> >>> I think the proposal involved using a ThreadPoolExecutor, which see= med >>> >>> to not quite work as well as what we have. =C2=A0I think it'll be e= asier to >>> >>> simply pass a global context that keeps a counter of the actively >>> >>> running threads, and pass that into each IW's CMS? >>> >>> >>> >>> On Thu, Apr 14, 2011 at 8:25 AM, Simon Willnauer >>> >>> wrote: >>> >>>> On Thu, Apr 14, 2011 at 5:20 PM, Jason Rutherglen >>> >>>> wrote: >>> >>>>> Today the ConcurrentMergeScheduler allows setting the max thread >>> >>>>> count and is bound to a single IndexWriter. >>> >>>>> >>> >>>>> However in the [common] case of multiple IndexWriters running in >>> >>>>> the same process, this disallows one from managing the aggregate >>> >>>>> number of merge threads executing at any given time. >>> >>>>> >>> >>>>> I think this can be fixed, shall I open an issue? >>> >>>> >>> >>>> go ahead! I think I have seen this suggestion somewhere maybe you >>> >>>> need >>> >>>> to see if there is one already >>> >>>> >>> >>>> simon >>> >>>>> >>> >>>>> >>> >>>>> -----------------------------------------------------------------= ---- >>> >>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org >>> >>>>> For additional commands, e-mail: dev-help@lucene.apache.org >>> >>>>> >>> >>>>> >>> >>>> >>> >>>> ------------------------------------------------------------------= --- >>> >>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org >>> >>>> For additional commands, e-mail: dev-help@lucene.apache.org >>> >>>> >>> >>>> >>> >>> >>> >>> -------------------------------------------------------------------= -- >>> >>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org >>> >>> For additional commands, e-mail: dev-help@lucene.apache.org >>> >>> >>> >>> >>> >> >>> >> >>> >> >>> >> -- >>> >> Kirill Zakharenko/=D0=9A=D0=B8=D1=80=D0=B8=D0=BB=D0=BB =D0=97=D0=B0= =D1=85=D0=B0=D1=80=D0=B5=D0=BD=D0=BA=D0=BE >>> >> E-Mail/Jabber: earwin@gmail.com >>> >> Phone: +7 (495) 683-567-4 >>> >> ICQ: 104465785 >>> >> >>> >> --------------------------------------------------------------------= - >>> >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org >>> >> For additional commands, e-mail: dev-help@lucene.apache.org >>> >> >>> >> >>> > >>> > --------------------------------------------------------------------- >>> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org >>> > For additional commands, e-mail: dev-help@lucene.apache.org >>> > >>> > >>> >>> >>> >>> -- >>> Kirill Zakharenko/=D0=9A=D0=B8=D1=80=D0=B8=D0=BB=D0=BB =D0=97=D0=B0=D1= =85=D0=B0=D1=80=D0=B5=D0=BD=D0=BA=D0=BE >>> E-Mail/Jabber: earwin@gmail.com >>> Phone: +7 (495) 683-567-4 >>> ICQ: 104465785 >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org >>> For additional commands, e-mail: dev-help@lucene.apache.org >>> >> >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org > For additional commands, e-mail: dev-help@lucene.apache.org > > --=20 Kirill Zakharenko/=D0=9A=D0=B8=D1=80=D0=B8=D0=BB=D0=BB =D0=97=D0=B0=D1=85= =D0=B0=D1=80=D0=B5=D0=BD=D0=BA=D0=BE E-Mail/Jabber: earwin@gmail.com Phone: +7 (495) 683-567-4 ICQ: 104465785 --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional commands, e-mail: dev-help@lucene.apache.org