From couchdb-user-return-739-apmail-incubator-couchdb-user-archive=incubator.apache.org@incubator.apache.org Sun Jul 13 16:53:19 2008 Return-Path: Delivered-To: apmail-incubator-couchdb-user-archive@locus.apache.org Received: (qmail 20671 invoked from network); 13 Jul 2008 16:53:19 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 13 Jul 2008 16:53:19 -0000 Received: (qmail 74671 invoked by uid 500); 13 Jul 2008 16:53:18 -0000 Delivered-To: apmail-incubator-couchdb-user-archive@incubator.apache.org Received: (qmail 74641 invoked by uid 500); 13 Jul 2008 16:53:18 -0000 Mailing-List: contact couchdb-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: couchdb-user@incubator.apache.org Delivered-To: mailing list couchdb-user@incubator.apache.org Received: (qmail 74630 invoked by uid 99); 13 Jul 2008 16:53:18 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Jul 2008 09:53:18 -0700 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.97.50.139] (HELO jan.prima.de) (83.97.50.139) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Jul 2008 16:52:25 +0000 Received: from [10.0.2.3] (e179134130.adsl.alicedsl.de [::ffff:85.179.134.130]) (AUTH: LOGIN jan, SSL: TLSv1/SSLv3,128bits,AES128-SHA) by jan.prima.de with esmtp; Sun, 13 Jul 2008 16:46:47 +0000 Message-Id: From: Jan Lehnardt To: couchdb-user@incubator.apache.org In-Reply-To: <5aea26870807130905h12f47a37i426b39fd9f4c17d1@mail.gmail.com> Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v928.1) Subject: Re: Practical storage limit Date: Sun, 13 Jul 2008 18:46:46 +0200 References: <5aea26870807130905h12f47a37i426b39fd9f4c17d1@mail.gmail.com> X-Mailer: Apple Mail (2.928.1) X-Virus-Checked: Checked by ClamAV on apache.org On Jul 13, 2008, at 18:05, Peter Eddy wrote: >> What I'd do is query the view for updating after each bulk insert. > > This is what I've been doing, however it seems less than ideal. It > means that the bulk insert code needs to know about all views that > have been defined. > > What I want to avoid, of course, is a user invoking a view and having > to wait a long time (minutes in some cases) for that view to be > updated in order to get their results. I would gladly trade document > insertion time for automatic view updates. > > It could be argued that bulk updates are a special case that happen > only in exceptional situations. But I can easily imagine normal > application functionality that might add many documents, and this code > would then need to think about how many documents were added and if > the views should be updated in the process in order to avoid long > user-time delays. The larger the application, the more messy and error > prone this becomes. > > Anyway, I've been meaning to mention this for a long time, so since it > came up... I'll make a FaQ out of this :-) To get on write view update semantics, you can write a little daemon script that runs alongside CouchDB and is specified in couch.ini as DbUpdateNotificationProcesses. This deamon gets sent a notification each time the database is changed and could in turn trigger a view update every N document inserts or every Y seconds, whichever occurs first. The reason not to integrate each doc as it comes in is that it is horribly inefficient and CouchDB is designed to do view index updates very fast, so batching is a good idea. To get a list of all views in a database, you can do a GET /db/_all_docs?startkey=_design/&endkey=_design/ZZZZ (we will have a /db/_all_design_docs view to make the ZZZZ-hack go away). That should solve your problem. Yes, such a daemon should be shipped with CouchDB, but we haven't got around to work on the deployment infrastructure yet. Any contributions to this are very welcome. I think the developer's choice of language for helper scripts is Python, but any will do, whatever suits you best. Cheers Jan -- The FaQ entry is at the bottom of this page. http://wiki.apache.org/couchdb/FrequentlyAskedQuestions BTW: Is there a FaQ module or whatever for MoinMoin? it would be nice to get a MediaWiki-like table of contents at the top. Noah?