From couchdb-user-return-2052-apmail-incubator-couchdb-user-archive=incubator.apache.org@incubator.apache.org Mon Dec 01 08:02:57 2008 Return-Path: Delivered-To: apmail-incubator-couchdb-user-archive@locus.apache.org Received: (qmail 98199 invoked from network); 1 Dec 2008 08:02:55 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 1 Dec 2008 08:02:55 -0000 Received: (qmail 54976 invoked by uid 500); 1 Dec 2008 08:03:06 -0000 Delivered-To: apmail-incubator-couchdb-user-archive@incubator.apache.org Received: (qmail 54944 invoked by uid 500); 1 Dec 2008 08:03:06 -0000 Mailing-List: contact couchdb-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: couchdb-user@incubator.apache.org Delivered-To: mailing list couchdb-user@incubator.apache.org Received: (qmail 54933 invoked by uid 99); 1 Dec 2008 08:03:05 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Dec 2008 00:03:05 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of antony.blakey@gmail.com designates 209.85.198.246 as permitted sender) Received: from [209.85.198.246] (HELO rv-out-0708.google.com) (209.85.198.246) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Dec 2008 08:01:37 +0000 Received: by rv-out-0708.google.com with SMTP id k29so2895300rvb.0 for ; Mon, 01 Dec 2008 00:02:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:from:to :in-reply-to:content-type:content-transfer-encoding:mime-version :subject:date:references:x-mailer; bh=F/yTdwACRR/NxoNxR7i2d2sosiuaqguMXFLedRRw0UI=; b=btxObjx3ghpoAvjiYvj62qvkyvYcKyF7kAjPmlbgchoRbJubCiSnyOJNJmCapQJ3PO aLK6Ujr3t0QxV4jyLUN62u+GrjBn0QRtB1BaZi6HZPOudf8ywpMVDqvbnXiEg0lFSJH7 CxvV6cZ3h5Era3rFTL/p2u/6/LisBAXv5wi6Q= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:from:to:in-reply-to:content-type :content-transfer-encoding:mime-version:subject:date:references :x-mailer; b=on0KdyXMg/s5KqD0R6/MIXwfPvLvPWeFFjgYGSS3Vp5EZnd/8TdYIVvOEPtLCwrEw0 7R7Si4sZJWPp3HGLts9Wt7hgKFqrLuYcUEVP1SEp1CyQFFAZ3qeT9Bb4/Dax6QnPOI+q 3aak0KZKjyaauiK5sB5R/GHTxBUbmt1Jj+6/A= Received: by 10.140.203.9 with SMTP id a9mr5109403rvg.64.1228118544420; Mon, 01 Dec 2008 00:02:24 -0800 (PST) Received: from ?192.168.0.19? (ppp121-45-41-103.lns10.adl2.internode.on.net [121.45.41.103]) by mx.google.com with ESMTPS id k37sm11246818rvb.1.2008.12.01.00.02.22 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 01 Dec 2008 00:02:23 -0800 (PST) Message-Id: <82A07D10-9016-4031-9913-F80584EC08AD@gmail.com> From: Antony Blakey To: couchdb-user@incubator.apache.org In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v929.2) Subject: Re: Ordering by values calculated by a map/reduce Date: Mon, 1 Dec 2008 18:32:19 +1030 References: X-Mailer: Apple Mail (2.929.2) X-Virus-Checked: Checked by ClamAV on apache.org On 01/12/2008, at 6:09 PM, Ben Bangert wrote: > I want to solve what I thought was a fairly simple problem, though > unfortunately it seems to be rather tricky. I asked on the IRC > channel, and got some good input, but neither really seemed like a > very good solution. > > The problem: > > I want to allow users to rate things. They do this very very > frequently, so I can't store it in the actual document being rated, > so I have Rating documents. It's easy enough to write a map/reduce > that gives me the computed average rating for a given document, > however, it seems to be impossible to get a listing of the highest > rated documents, as I can only get the computed rating for a > document one at a time. > > The possible solutions: > - Buffer rating additions, then at a later time, run through them > and calculate the new average rating, store it in the document as > computed_rating, so I can order on that in the key > - Cron a job that goes in and looks for new rating every 5 mins or > whatever, and then does the same as the previous solution by storing > it in a computed_rating field > > I'm not a fan of either of these, because #1 means if my webapp > hiccups, I lose ratings, and #2 is just a pain to keep sweeping the > db for new Rating documents then going through updating all the > documents. > > Is there really no other solutions that don't require me to store > the computed rating in the doc itself? There's no way I can perhaps > order on the value from the map/reduce, rather than only being able > to order on the key? I use an external query handler to solve problems that don't fit map/ reduce. I've modified Paul Davis's _external handler to pass the current update_seq whenever an external query is made. In the external process (ruby in my case) I maintain a SQLite database that I can use for queries that Couch isn't suited for. Whenever a query comes in, I compare the supplied update_seq with one stored in my sqlite db (of course I cache that in memory as long as the process lives). If the sqlite db is out of date, then I do a _all_docs_by_seq and update sqlite, including the update_seq record, before doing the (SQL) query and responding with a JSON document in the same format as Couch would. I can delete the sqlite db at any time (well, while the process is stopped) because it will get recreated/updated when a query comes in as necessary. This works with replication. This system has the same lazy-update characteristics as Couch views, and has the additional advantage that you can do in-memory caching in the external process which depending on your update frequency, means you rarely hit the db. Antony Blakey -------------------------- CTO, Linkuistics Pty Ltd Ph: 0438 840 787 What can be done with fewer [assumptions] is done in vain with more -- William of Ockham (ca. 1285-1349)