From user-return-16309-apmail-couchdb-user-archive=couchdb.apache.org@couchdb.apache.org Mon May 23 05:56:30 2011 Return-Path: X-Original-To: apmail-couchdb-user-archive@www.apache.org Delivered-To: apmail-couchdb-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9FB224D3E for ; Mon, 23 May 2011 05:56:30 +0000 (UTC) Received: (qmail 76651 invoked by uid 500); 23 May 2011 05:56:29 -0000 Delivered-To: apmail-couchdb-user-archive@couchdb.apache.org Received: (qmail 76579 invoked by uid 500); 23 May 2011 05:56:28 -0000 Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@couchdb.apache.org Delivered-To: mailing list user@couchdb.apache.org Received: (qmail 76569 invoked by uid 99); 23 May 2011 05:56:27 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 05:56:27 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of afters.mail@gmail.com designates 209.85.214.180 as permitted sender) Received: from [209.85.214.180] (HELO mail-iw0-f180.google.com) (209.85.214.180) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 05:56:21 +0000 Received: by iwn6 with SMTP id 6so6799162iwn.11 for ; Sun, 22 May 2011 22:55:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=EfXIjUsfZ14yaZahgN9LnP+tJVlBy40c7ClMFtEOEt0=; b=gLFOMIJzz8cdarukbk7NTzUDgc6SH0DlijE6JeKQxXHWqqMZcSU8zP4cEl6imjhpIX KuNzRLiYk9MUBDDmG0Nk+RMrCl+UlYL/tMdfWbwW0O53q+O+OP6CBj64Q1zix2LTXHiO P/y6xiuYOkJl4LdiPIoBx6cj31j5jJDNG/+O0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=i3PdsQYOF9YLpymYKd5GtMaUtd6Don8vZjgnI2PeBTmQKs1xU7OCm6nONibybKJs+W rx/xdBImoLWXaKLmaJ8mxSuR7e63rHngx0SjKPKbdM1Unk6Ok0Z8YLBgK7P17wjstnNT T5sEE5puj2fbT0k+9S935MRiyCP7D66EdFFb4= MIME-Version: 1.0 Received: by 10.231.193.233 with SMTP id dv41mr1835445ibb.186.1306130159770; Sun, 22 May 2011 22:55:59 -0700 (PDT) Received: by 10.231.35.75 with HTTP; Sun, 22 May 2011 22:55:59 -0700 (PDT) In-Reply-To: References: Date: Mon, 23 May 2011 08:55:59 +0300 Message-ID: Subject: Re: DB that mirrors a view From: afters To: user@couchdb.apache.org Content-Type: multipart/alternative; boundary=0016e68ee14a341c3804a3eb202b X-Virus-Checked: Checked by ClamAV on apache.org --0016e68ee14a341c3804a3eb202b Content-Type: text/plain; charset=ISO-8859-1 Jason, Thanks for the detailed answer! I've actually written code along the same lines as yours (only with eval instead of 'new Function' - thanks for the tip). I'm actually interested in the reduce results, rather than the map results. It's still possible to deduce the affected reduce-keys for each change, just by using the map function: just clip the emitted map key according to the desired group_level. Alas, it's not possible to deduce the query results - as reduce depends also on previous changes. If only I could prevent the view from being updated for a little while, I could read the changes feed till I'm in line with the view's seqnum... Benoit, if you're reading this I'd love to hear more about your fork. a. On 23 May 2011 04:25, Jason Smith wrote: > I think Benoit has a fork of CouchDB to do this. Also I believe this > feature (a _changes filter that only shows docs that change a view) is > possibly coming in the future. > > As a side-note, view queries support an ?update_seq=true parameter, > which will tell you the seqnum of the database which that view > represents. > > Anyway, probably the best bang-for-buck with NodeJS is simply to > download the design document and run the map() functions yourself. > They are purely functional, and they have no (or perhaps well-known) > side-effects. > > The basic idea is: > > 1. Fetch the design document as JSON > 2. Extract the .views.whatever.map value as a string > 3. Build a wrapper map() function that has a custom emit() > 4. Run it for every _changes update > > Below is my basic idea. I think I've seen it elsewhere, perhaps in > Benoit's fork. But this is with Node. It could use some optimization > too (don't re-define map() every function call). > > var ddoc = get_ddoc(); > var couch_map = ddoc.views.someview.map; > > var changed_keys = []; > function emit(key, val) { > console.log("Emit! " + JSON.stringify(key)); > changed_keys.push(key); > } > > var map = new Function("doc, emit", "var couch_map = " + couch_map + > "; couch_map(doc);); > > changes.forEach(function(change) { > map(change.doc, emit); > }) > > console.log("These keys were emitted: " + JSON.stringify(changed_keys)); > > > Hi guys, > > > > I'm giving a serious look into building a db that mirrors the contents of > a > > view (for running map on the results of map-reduce). > > > > The hurdle that I'm facing is tracking changes in the view: > > It's possible to listen to the db's changes feed and deduce which > view-keys > > should be updated for a particular seqnum, then query the view with these > > keys. Unfortunately, the view would only return the latest results, that > may > > easily incorporate changes from later seqnums, thus making my mirroring > db > > inconsistent with the view. > > > > Any thoughts? Has anyone else meddled with turning a view to a db? > > It would be useful if I could somehow freeze the update of the view (I > guess > > I could do it by replicating its db only to a certain point, but that > would > > mean a whole other db) > > > > Thanks, > > a. > > > > > > -- > Iris Couch > --0016e68ee14a341c3804a3eb202b--