Return-Path: Delivered-To: apmail-couchdb-dev-archive@www.apache.org Received: (qmail 52419 invoked from network); 1 Jun 2010 14:11:13 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 1 Jun 2010 14:11:13 -0000 Received: (qmail 30599 invoked by uid 500); 1 Jun 2010 14:11:13 -0000 Delivered-To: apmail-couchdb-dev-archive@couchdb.apache.org Received: (qmail 30511 invoked by uid 500); 1 Jun 2010 14:11:12 -0000 Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@couchdb.apache.org Delivered-To: mailing list dev@couchdb.apache.org Received: (qmail 30502 invoked by uid 99); 1 Jun 2010 14:11:12 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 14:11:12 +0000 X-ASF-Spam-Status: No, hits=-2.3 required=10.0 tests=RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [145.58.30.185] (HELO out1b.mail.omroep.nl) (145.58.30.185) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 14:11:01 +0000 Received: from localhost (ou1bclean [10.10.30.159]) by out1b.mail.omroep.nl (Postfix MTA - NPO ICT) with ESMTP id ADF7E300040C for ; Tue, 1 Jun 2010 16:10:41 +0200 (CEST) X-Virus-Scanned: NPO ICT Received: from tweehoog.vpro.nl (tweehoog.vpro.nl [145.58.169.4]) by out1b.mail.omroep.nl (Postfix MTA - NPO ICT) with ESMTP id 956043000137 for ; Tue, 1 Jun 2010 16:10:41 +0200 (CEST) Received: from exmail.vpro.nl ([145.58.171.81] helo=VS-EX-01.intra.vpro.nl) by tweehoog.vpro.nl with esmtp (Exim 3.36 #1) id 1OJSAz-0000VT-00 for dev@couchdb.apache.org; Tue, 01 Jun 2010 16:10:41 +0200 Received: from [145.58.169.113] (145.58.169.113) by VS-EX-01.intra.vpro.nl (145.58.171.81) with Microsoft SMTP Server (TLS) id 8.2.247.2; Tue, 1 Jun 2010 16:10:40 +0200 Message-ID: <4C0514E0.2010006@vpro.nl> Date: Tue, 1 Jun 2010 16:10:40 +0200 From: Nils Breunese User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.9) Gecko/20100423 Thunderbird/3.0.4 MIME-Version: 1.0 To: Subject: Performance issues with external process Content-Type: text/plain; charset="ISO-8859-1"; format=flowed Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org Hello all, Last weekend we ran a mashup site [0] for the Dutch Pinkpop music festival. A backend harvesting process stored all data in CouchDB and we used couchdb-lucene for full-text indexing. (There are a lot of other moving parts, it's not all run on CouchDB.) All individual parts of our setup performed very well, but still we had some serious performance problems which seem to trace back to the fact that CouchDB and external processes communicate via stdin/stdout (a pipe), which AFAIK is a communication channel that does not allow for concurrency. Queries to CouchDB were fast, query times on couchdb-lucene were fast, but still the end result was slow, because queries to couchdb-lucene all needed to be serialized and go through the pipe to couchdb-lucene's handler script. We have discussed this with Robert Newson of couchdb-lucene and he suggested going around CouchDB and talking HTTP to couchdb-lucene directly. While this may work, I thought I'd join the dev list and bring up this issue on the dev list here and ask if there might be a way to allow concurrent access to external processes somehow, because this was a performance bottleneck we hadn't accounted for and I feel others may run into this as well at some point. Nils. [0] http://pinkpop.vpro.nl/ De informatie vervat in deze e-mail en meegezonden bijlagen is uitsluitend= bedoeld voor gebruik door de geadresseerde en kan vertrouwelijke informati= e bevatten. Openbaarmaking, vermenigvuldiging, verspreiding en/of verstrekk= ing van deze informatie aan derden is voorbehouden aan geadresseerde. De VP= RO staat niet in voor de juiste en volledige overbrenging van de inhoud van= een verzonden e-mail, noch voor tijdige ontvangst daarvan.