couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From J Chris Anderson <jch...@apache.org>
Subject Re: querying multiple views
Date Thu, 08 Jul 2010 18:40:26 GMT

On Jul 8, 2010, at 10:43 AM, Norman Barker wrote:

> Hi,
> 
> I have been thinking about how to query multiple views at one time.
> 
> I have an erlang handler in couchdb that takes a http post containing
> N view queries, each query contains a startkey and an endkey, I then
> open up each view in parallel (using pmap) and accumulate the doc ids,
> then I use the erlang sets module to get the unique values. All good
> and looks pretty (and works), though it doesn't scale since I am
> holding all the results on the server (potential memory overload!)
> whereas I would like to stream the results to the client one by one.
> 
> I am thinking of doing the following but have some questions;
> 
> My first question is when I do
> 
> couch_view:fold(View, FoldlFun, FoldAccInit,
> couch_httpd_view:make_key_options(Args)),
> 
> is there a way to call the _count reduce function in code to find the
> number of rows in the slice between startkey and endkey?
> 
> If so, I would like to order all the views in the posted query
> document by the result of _count from smallest to largest.
> 
> I would then fold over the smallest result view and pull each document
> id (*) in turn.
> 
> With each document id I would then call each of the other views in
> turn with their startkey and endkey and in addition include
> startkey_docid and endkey_docid with the docid in * above, again
> calling _count I can test for inclusion. If the doc id is in all views
> then I will immediate stream this to the client.
> 
> Am I doing something stupid, is this optimal?
> 

It sounds like you are on the right track. this could be a very valuable patch to CouchDB
once you have it working.

> Any help with the programmatic _count call would be great.
> 

One hint: maybe the call to reduce_to_count will help.

Here's an implementation of a reduce query in Erlang.

http://github.com/jchris/hovercraft/blob/master/hovercraft.erl#L217

Sorry I can't be more helpful. I've successfully bootstrapped this stuff in my head before,
but it always takes a couple of hours of turning my brain into a step debugger.

Good luck!

Once you get deeper into the code you might have better luck getting responses on the dev@
list or maybe the #couchdb IRC channel on freenode.

Chris


> thanks,
> 
> Norman


Mime
View raw message