Return-Path: Delivered-To: apmail-lucene-dev-archive@www.apache.org Received: (qmail 88867 invoked from network); 28 Sep 2010 14:33:01 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 28 Sep 2010 14:33:01 -0000 Received: (qmail 60161 invoked by uid 500); 28 Sep 2010 14:33:00 -0000 Delivered-To: apmail-lucene-dev-archive@lucene.apache.org Received: (qmail 58981 invoked by uid 500); 28 Sep 2010 14:32:57 -0000 Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@lucene.apache.org Delivered-To: mailing list dev@lucene.apache.org Received: (qmail 58968 invoked by uid 99); 28 Sep 2010 14:32:56 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Sep 2010 14:32:56 +0000 X-ASF-Spam-Status: No, hits=0.7 required=10.0 tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Sep 2010 14:32:51 +0000 Received: by wyb34 with SMTP id 34so7391589wyb.35 for ; Tue, 28 Sep 2010 07:32:30 -0700 (PDT) MIME-Version: 1.0 Received: by 10.216.10.145 with SMTP id 17mr35135wev.27.1285684349929; Tue, 28 Sep 2010 07:32:29 -0700 (PDT) Received: by 10.216.70.135 with HTTP; Tue, 28 Sep 2010 07:32:29 -0700 (PDT) In-Reply-To: <35030446-B726-4141-AF52-FE641C121E48@cominvent.com> References: <35030446-B726-4141-AF52-FE641C121E48@cominvent.com> Date: Tue, 28 Sep 2010 10:32:29 -0400 Message-ID: Subject: Re: Indexing and threads From: Michael McCandless To: dev@lucene.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable I can't speak to how Solr handles threads, but in Lucene the two docs are indexed concurrently. Internally, Lucene's IndexWriter has separate thread states that hold the RAM buffer of the inverted docs. Multiple threads work on separate thread states concurrently. The one big exception to this is flushing a new segment, which is currently single threaded and can be quite a bottleneck (I wrote about this problem at http://chbits.blogspot.com/2010/09/lucenes-indexing-is-fast.html). Mike On Tue, Sep 28, 2010 at 9:32 AM, Jan H=F8ydahl / Cominvent wrote: > Hi, > > How are threads being used when indexing? > > Let's say document A and B are ingested in parallell to XMLUpdateRequestH= andler in two separate threads. > How far down the chain are the processing of these done in the two separa= te threads? > Is the full UpdateRequestChain run in the same thread as the incoming req= uest? > Is analysis done in the request thread or in a single indexing thread? > Are ADDs added to the same "commit queue", and then from COMMIT and down = to Lucene segment building everything is single-threaded? > > -- > Jan H=F8ydahl, search solution architect > Cominvent AS - www.cominvent.com > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org > For additional commands, e-mail: dev-help@lucene.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional commands, e-mail: dev-help@lucene.apache.org