Return-Path: Delivered-To: apmail-lucene-java-dev-archive@www.apache.org Received: (qmail 62466 invoked from network); 1 Oct 2009 21:02:58 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 1 Oct 2009 21:02:58 -0000 Received: (qmail 5156 invoked by uid 500); 1 Oct 2009 21:02:58 -0000 Delivered-To: apmail-lucene-java-dev-archive@lucene.apache.org Received: (qmail 5085 invoked by uid 500); 1 Oct 2009 21:02:57 -0000 Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-dev@lucene.apache.org Delivered-To: mailing list java-dev@lucene.apache.org Received: (qmail 5077 invoked by uid 99); 1 Oct 2009 21:02:57 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Oct 2009 21:02:57 +0000 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [209.85.217.226] (HELO mail-gx0-f226.google.com) (209.85.217.226) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Oct 2009 21:02:48 +0000 Received: by gxk26 with SMTP id 26so669775gxk.11 for ; Thu, 01 Oct 2009 14:01:27 -0700 (PDT) MIME-Version: 1.0 Received: by 10.150.233.3 with SMTP id f3mr416031ybh.148.1254430881688; Thu, 01 Oct 2009 14:01:21 -0700 (PDT) In-Reply-To: <59b3eb370910011156l538d966chc26222d40970a20b@mail.gmail.com> References: <25697034.post@talk.nabble.com> <359a92830910010847x5f9d8812o6cb95f9877b91e3f@mail.gmail.com> <4AC4D06F.5050304@gmail.com> <59b3eb370910010949l488c473akda58ebe50ca3bdd2@mail.gmail.com> <9ac0c6aa0910011001p2e5645bdp2c65b3c65d8cf60e@mail.gmail.com> <59b3eb370910011156l538d966chc26222d40970a20b@mail.gmail.com> Date: Thu, 1 Oct 2009 17:01:21 -0400 Message-ID: <9ac0c6aa0910011401y72d16407tc137ce4a135e7c29@mail.gmail.com> Subject: Re: Optimization and Corruption Issues From: Michael McCandless To: java-dev@lucene.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org On Thu, Oct 1, 2009 at 2:56 PM, Earwin Burrfoot wrote: >>> If you're doing a full optimize, you're still using a single thread. Am= I wrong? >> >> Depends on how many merges are required, and, the merge scheduler. =A0In >> this case (w/ 7000 segments, which is way too many, normally!), >> assuming ConcurrentMergeScheduler, multiple threads will be used since >> many merges will be pending. >> >> When it gets down to the last (enormous) merge, it's only one thread. > I'm speaking about full optimize. Is there any way to do it more > efficiently then running a single, last (enormous) merge? I guess we could merge different parts of the index w/ different threads, if we wanted to push concurrency down into a single merge. > If you try to parallelize, you're merging some documents several times > (more work) and killing your disks, as merges are mostly IO-bound. Actually I've found merging of the postings to be CPU bound. I think the priority queue, and decode/encode of vInt, are the big CPU costs. Mike --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org For additional commands, e-mail: java-dev-help@lucene.apache.org