From user-return-12331-apmail-couchdb-user-archive=couchdb.apache.org@couchdb.apache.org Thu Aug 26 15:00:56 2010 Return-Path: Delivered-To: apmail-couchdb-user-archive@www.apache.org Received: (qmail 45344 invoked from network); 26 Aug 2010 15:00:56 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 26 Aug 2010 15:00:56 -0000 Received: (qmail 96526 invoked by uid 500); 26 Aug 2010 15:00:54 -0000 Delivered-To: apmail-couchdb-user-archive@couchdb.apache.org Received: (qmail 96348 invoked by uid 500); 26 Aug 2010 15:00:52 -0000 Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@couchdb.apache.org Delivered-To: mailing list user@couchdb.apache.org Received: (qmail 96340 invoked by uid 99); 26 Aug 2010 15:00:51 -0000 Received: from Unknown (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 15:00:51 +0000 X-ASF-Spam-Status: No, hits=0.7 required=10.0 tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [74.125.83.180] (HELO mail-pv0-f180.google.com) (74.125.83.180) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 15:00:42 +0000 Received: by pvc30 with SMTP id 30so1117304pvc.11 for ; Thu, 26 Aug 2010 08:00:20 -0700 (PDT) Received: by 10.114.89.11 with SMTP id m11mr11320025wab.150.1282834819505; Thu, 26 Aug 2010 08:00:19 -0700 (PDT) Received: from [10.0.1.2] (c-24-130-240-73.hsd1.ca.comcast.net [24.130.240.73]) by mx.google.com with ESMTPS id c24sm4606530wam.19.2010.08.26.08.00.17 (version=TLSv1/SSLv3 cipher=RC4-MD5); Thu, 26 Aug 2010 08:00:18 -0700 (PDT) Sender: J Chris Anderson Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Apple Message framework v1081) Subject: Re: Seamless view rebuilding? From: J Chris Anderson In-Reply-To: <4C767D8F.7000704@gmail.com> Date: Thu, 26 Aug 2010 08:00:15 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: <4C766ECA.1090100@gmail.com> <726DB993-42A3-41E4-9F4A-D7D10D8A8AC1@apache.org> <4C767D8F.7000704@gmail.com> To: user@couchdb.apache.org X-Mailer: Apple Mail (2.1081) X-Virus-Checked: Checked by ClamAV on apache.org On Aug 26, 2010, at 7:43 AM, Patrick Barnes wrote: > What about a staging db? >=20 > If I had: > * Continuous replication from production->staging (for those piecemeal = updates) > * Made major batch updates to the staging server > * Rebuilt the staging server views, then > * Replicated from staging->production >=20 View files don't replicate, so this won't help for the batches. Best bet is probably to query the view periodically during the batch = import. Chris > Would the production db's views take as long to rebuild as the = single-db model, or does it have some mechanism to optimise it? >=20 > The couchdb server is also behind a proxy, so there might be some = solution there? >=20 > -PB >=20 >=20 > On 26/08/2010 11:48 PM, Adam Kocoloski wrote: >> But he doesn't have a new view, just a very large batch of updates = added to an existing view. >>=20 >> On Aug 26, 2010, at 9:46 AM, Robert Newson wrote: >>=20 >>> Create a new ddoc with your new view, query that view, waiting for = it >>> to build, and then copy your new ddoc over your old one. View = indexes >>> are named on disk after their digest specifically to allow this >>> offline building feature. :) >>>=20 >>> B. >>>=20 >>> On Thu, Aug 26, 2010 at 2:40 PM, Patrick Barnes = wrote: >>>> I have a database serving documents through a number of = intermediary >>>> application servers, to the users' web browsers. >>>>=20 >>>> There are two mechanisms by which documents are modified; >>>> a) Piecemeal updates as a result of user actions. (ie adding or = updating a >>>> record) >>>> b) Bulk updates, typically from import scripts, that might modify = tens of >>>> thousands of documents at once. >>>>=20 >>>> The problem I'm having, is that when a set of bulk updates go = through, it >>>> can take a long time to rebuild the view indexes. Meanwhile, = several user's >>>> web requests will time out until rebuilding is complete. >>>>=20 >>>> Stale=3Dok is a simple solution to the bulk problem, but the = application >>>> servers will also expect to be able to update documents, and = retrieve the >>>> changes immediately after. >>>>=20 >>>> Is there a good way to avoid these large view update delays? >>>>=20 >>>> -Patrick Barnes >>>>=20 >>=20 >>=20