From user-return-11728-apmail-couchdb-user-archive=couchdb.apache.org@couchdb.apache.org Tue Jul 27 21:37:40 2010 Return-Path: Delivered-To: apmail-couchdb-user-archive@www.apache.org Received: (qmail 81659 invoked from network); 27 Jul 2010 21:37:39 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 27 Jul 2010 21:37:39 -0000 Received: (qmail 99031 invoked by uid 500); 27 Jul 2010 21:37:38 -0000 Delivered-To: apmail-couchdb-user-archive@couchdb.apache.org Received: (qmail 98941 invoked by uid 500); 27 Jul 2010 21:37:37 -0000 Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@couchdb.apache.org Delivered-To: mailing list user@couchdb.apache.org Received: (qmail 98931 invoked by uid 99); 27 Jul 2010 21:37:37 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 21:37:37 +0000 X-ASF-Spam-Status: No, hits=0.7 required=10.0 tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [209.85.212.180] (HELO mail-px0-f180.google.com) (209.85.212.180) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 21:37:31 +0000 Received: by pxi3 with SMTP id 3so1218861pxi.11 for ; Tue, 27 Jul 2010 14:37:10 -0700 (PDT) Received: by 10.142.204.17 with SMTP id b17mr10860266wfg.142.1280266629983; Tue, 27 Jul 2010 14:37:09 -0700 (PDT) Received: from [192.168.1.102] (c-98-248-172-14.hsd1.ca.comcast.net [98.248.172.14]) by mx.google.com with ESMTPS id 33sm6162473wfg.9.2010.07.27.14.37.09 (version=TLSv1/SSLv3 cipher=RC4-MD5); Tue, 27 Jul 2010 14:37:09 -0700 (PDT) Sender: J Chris Anderson Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Apple Message framework v1081) Subject: Re: external handlers are not in sync with commits From: J Chris Anderson In-Reply-To: Date: Tue, 27 Jul 2010 14:37:07 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: To: user@couchdb.apache.org X-Mailer: Apple Mail (2.1081) On Jul 27, 2010, at 2:30 PM, Norman Barker wrote: > Hi, >=20 > I have written couchdb-clucene > (http://github.com/normanb/couchdb-clucene) and am doing a lot of > testing with heavy datasets where I am sending a bulk doc request with > 10 docs at a time, a couple of these every second for a couple of > minutes. >=20 > Very quickly couchdb backs up and hogs the cpu since the database > commit and return doesn't wait for an external handler to do its job. > The model of fire and forget is fine and I like it, very similar to > JMS, however since the external process is a singleton it has to be > very quick to keep up with load or the system slowly backs up. >=20 > Is there a way to either define a pool of externals, or to change the > default behaviour from fire and forget? >=20 Yes, but involves feature development for CouchDB. Essentially we need = the externals protocol to be non-blocking. There is a thread on dev@ = that touches on this. I'm not sure who wants to own making the patch, = but the technical requirements are pretty well known. Thank you for working on something so awesome! Chris > thanks, >=20 > Norman