From user-return-23094-apmail-couchdb-user-archive=couchdb.apache.org@couchdb.apache.org Wed Dec 12 22:14:46 2012 Return-Path: X-Original-To: apmail-couchdb-user-archive@www.apache.org Delivered-To: apmail-couchdb-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D3B55D5A9 for ; Wed, 12 Dec 2012 22:14:46 +0000 (UTC) Received: (qmail 90172 invoked by uid 500); 12 Dec 2012 22:14:45 -0000 Delivered-To: apmail-couchdb-user-archive@couchdb.apache.org Received: (qmail 90133 invoked by uid 500); 12 Dec 2012 22:14:45 -0000 Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@couchdb.apache.org Delivered-To: mailing list user@couchdb.apache.org Received: (qmail 90125 invoked by uid 99); 12 Dec 2012 22:14:45 -0000 Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Dec 2012 22:14:45 +0000 Received: from localhost (HELO mail-vc0-f180.google.com) (127.0.0.1) (smtp-auth username rnewson, mechanism plain) by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Dec 2012 22:14:45 +0000 Received: by mail-vc0-f180.google.com with SMTP id p16so1393813vcq.11 for ; Wed, 12 Dec 2012 14:14:43 -0800 (PST) MIME-Version: 1.0 Received: by 10.58.55.198 with SMTP id u6mr1652582vep.35.1355350483476; Wed, 12 Dec 2012 14:14:43 -0800 (PST) Received: by 10.52.68.209 with HTTP; Wed, 12 Dec 2012 14:14:43 -0800 (PST) In-Reply-To: References: <20121212210315.GE22667@translab.its.uci.edu> Date: Wed, 12 Dec 2012 22:14:43 +0000 Message-ID: Subject: Re: Options for Iterative Map Reduce From: Robert Newson To: "user@couchdb.apache.org" Content-Type: text/plain; charset=ISO-8859-1 Yes, our chained map-reduce is incremental. On 12 December 2012 22:07, nicholas a. evans wrote: > On Wed, Dec 12, 2012 at 4:03 PM, James Marca > wrote: >> I feel your pain but cannot offer any help. I also use your option 5: >> I use node.js to manually store view output into a separate db, with >> the doc _ids equal to the key of the view output, so that I can limit >> updates to only those things that change. > > Thanks James. Do you apply the changes incrementally, and if so how > do you detect which view rows (in the source DB) have changes so you > don't need to download the whole reduced/grouped view? And to the > point of my last email, how do you detect missing view rows in the > source DB and delete them from the chained DB? > >> This is one feature I really want in CouchDB...the ability to reduce >> view output and/or treat view output like a regular couchdb database. >> >> Like a flag or something that says, hey couchdb, save this view as a >> db automatically, but also update it when the source db changes (not >> on view, but on edit) > > Agreed. As far as I can tell, that's basically the Cloudant "dbcopy" > feature. I'm assuming their dbcopy updates incrementally. On the > other hand, the most common use case is probably alternate sort > functions. In that case, copying the view to a new DB seems wasteful > (of time, disk space, and programmer time), when sort_by functions > which generate secondary indexes should do the trick (and would be > easily kept in sync with the view they are indexing). > > I wish I knew enough Erlang to dig in and see if either secondary > indexes or a view changes feed is nasty complicated or trivial. > > Hey, I spent the last two weeks learning Lua in order to do some > complicated Redis scripting. How hard can Erlang and CouchDB > internals be? ;-) > > -- > Nick