couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Norman Barker <norman.bar...@gmail.com>
Subject Re: querying multiple views
Date Sun, 11 Jul 2010 14:27:47 GMT
Afshin

I have got the all clear from my work to release this as a patch, I
expect to be putting something up on github by the end of the week
(internal paperwork permitting). I am going to implement it as an
external handler so it can be used and reviewed and from there it will
be under a do what you want with it license so it can go into couchdb
if accepted.

Chris, thanks for the help with the reduce function and confirming the concept!

Norman

On Sun, Jul 11, 2010 at 8:02 AM, afshin afzali <a.afzali2003@gmail.com> wrote:
> Hi Norman, Chris
>
> I just wanted to say this is the same problem we are currently facing
> with. We are implementing a Local Business Directory application on
> couchdb. Our searches need to combine several keys together to find
> right entries. To do something like that in server side, we had the
> paging mechanism problem, so we have chosen that do Norman's algorithm
> in client side! I'll appreciate if there will be a successful progress
> in this issue.
>
> BEST,
> -- afshin
>
> On 7/8/10, J Chris Anderson <jchris@apache.org> wrote:
>>
>> On Jul 8, 2010, at 10:43 AM, Norman Barker wrote:
>>
>>> Hi,
>>>
>>> I have been thinking about how to query multiple views at one time.
>>>
>>> I have an erlang handler in couchdb that takes a http post containing
>>> N view queries, each query contains a startkey and an endkey, I then
>>> open up each view in parallel (using pmap) and accumulate the doc ids,
>>> then I use the erlang sets module to get the unique values. All good
>>> and looks pretty (and works), though it doesn't scale since I am
>>> holding all the results on the server (potential memory overload!)
>>> whereas I would like to stream the results to the client one by one.
>>>
>>> I am thinking of doing the following but have some questions;
>>>
>>> My first question is when I do
>>>
>>> couch_view:fold(View, FoldlFun, FoldAccInit,
>>> couch_httpd_view:make_key_options(Args)),
>>>
>>> is there a way to call the _count reduce function in code to find the
>>> number of rows in the slice between startkey and endkey?
>>>
>>> If so, I would like to order all the views in the posted query
>>> document by the result of _count from smallest to largest.
>>>
>>> I would then fold over the smallest result view and pull each document
>>> id (*) in turn.
>>>
>>> With each document id I would then call each of the other views in
>>> turn with their startkey and endkey and in addition include
>>> startkey_docid and endkey_docid with the docid in * above, again
>>> calling _count I can test for inclusion. If the doc id is in all views
>>> then I will immediate stream this to the client.
>>>
>>> Am I doing something stupid, is this optimal?
>>>
>>
>> It sounds like you are on the right track. this could be a very valuable
>> patch to CouchDB once you have it working.
>>
>>> Any help with the programmatic _count call would be great.
>>>
>>
>> One hint: maybe the call to reduce_to_count will help.
>>
>> Here's an implementation of a reduce query in Erlang.
>>
>> http://github.com/jchris/hovercraft/blob/master/hovercraft.erl#L217
>>
>> Sorry I can't be more helpful. I've successfully bootstrapped this stuff in
>> my head before, but it always takes a couple of hours of turning my brain
>> into a step debugger.
>>
>> Good luck!
>>
>> Once you get deeper into the code you might have better luck getting
>> responses on the dev@ list or maybe the #couchdb IRC channel on freenode.
>>
>> Chris
>>
>>
>>> thanks,
>>>
>>> Norman
>>
>>
>

Mime
View raw message