incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Norman Barker <norman.bar...@gmail.com>
Subject Re: querying multiple views
Date Thu, 22 Jul 2010 23:27:46 GMT
ok, I have this working in practise, though there are still many
improvements that could be made. Sorry for the delay, 1.0
experimenting took some of my time last week (yay for 1.0!!).

Since couchdb now has an eccn number and ssl components this will take
me a little longer to release this code, but it will go out for
comment shortly.

I did sort the views from smallest to largest (by seeing the number of
hits in each view) and the I used the process ring example where you
send a message in a ring, each node in the ring represents a view and
if there is an intersection it sends a message to the next and if it
gets back to the start then it is sent as a chunk to the user. If the
match fails then the ring stops (actually it sends an empty message
back to the main node).

Interesting problem, but I am sure this is just a start, handling
externals which return all their data in one blob (as they don't
stream) is also an issue.

I will work on getting the code out.

Norman

On Sun, Jul 11, 2010 at 11:13 AM, afshin afzali <a.afzali2003@gmail.com> wrote:
> Thank You Very Much Guys.
>
> -- afshin
>
> On Sun, Jul 11, 2010 at 7:27 PM, J Chris Anderson <jchris@gmail.com> wrote:
>
>>
>> On Jul 11, 2010, at 7:27 AM, Norman Barker wrote:
>>
>> > Afshin
>> >
>> > I have got the all clear from my work to release this as a patch, I
>> > expect to be putting something up on github by the end of the week
>> > (internal paperwork permitting). I am going to implement it as an
>> > external handler so it can be used and reviewed and from there it will
>> > be under a do what you want with it license so it can go into couchdb
>> > if accepted.
>> >
>>
>> If you're implementing in Erlang, you probably don't need to go full hog as
>> an external.
>>
>> Because the configuration system is so modular, you are probably best
>> adding it as a new httpd_design_handler, or httpd_db_handler.
>>
>> It should be easy to create a new module and link it in via the
>> configuration file.
>>
>> I don't know if you plan to allow querying across databases (I'd suggest
>> restricting the queries to a single database if you want to stay within
>> CouchDB's security model, and have it more likely to be accepted as a
>> patch.)
>>
>> We should really move this discussion to dev@ -- a lot of the developers
>> only give a cursory glance at the user list, so you will get more valuable
>> feedback there.
>>
>> Thanks for taking the time to write and release the patch!
>>
>> Chris
>>
>> > Chris, thanks for the help with the reduce function and confirming the
>> concept!
>> >
>> > Norman
>> >
>> > On Sun, Jul 11, 2010 at 8:02 AM, afshin afzali <a.afzali2003@gmail.com>
>> wrote:
>> >> Hi Norman, Chris
>> >>
>> >> I just wanted to say this is the same problem we are currently facing
>> >> with. We are implementing a Local Business Directory application on
>> >> couchdb. Our searches need to combine several keys together to find
>> >> right entries. To do something like that in server side, we had the
>> >> paging mechanism problem, so we have chosen that do Norman's algorithm
>> >> in client side! I'll appreciate if there will be a successful progress
>> >> in this issue.
>> >>
>> >> BEST,
>> >> -- afshin
>> >>
>> >> On 7/8/10, J Chris Anderson <jchris@apache.org> wrote:
>> >>>
>> >>> On Jul 8, 2010, at 10:43 AM, Norman Barker wrote:
>> >>>
>> >>>> Hi,
>> >>>>
>> >>>> I have been thinking about how to query multiple views at one time.
>> >>>>
>> >>>> I have an erlang handler in couchdb that takes a http post containing
>> >>>> N view queries, each query contains a startkey and an endkey, I
then
>> >>>> open up each view in parallel (using pmap) and accumulate the doc
ids,
>> >>>> then I use the erlang sets module to get the unique values. All
good
>> >>>> and looks pretty (and works), though it doesn't scale since I am
>> >>>> holding all the results on the server (potential memory overload!)
>> >>>> whereas I would like to stream the results to the client one by
one.
>> >>>>
>> >>>> I am thinking of doing the following but have some questions;
>> >>>>
>> >>>> My first question is when I do
>> >>>>
>> >>>> couch_view:fold(View, FoldlFun, FoldAccInit,
>> >>>> couch_httpd_view:make_key_options(Args)),
>> >>>>
>> >>>> is there a way to call the _count reduce function in code to find
the
>> >>>> number of rows in the slice between startkey and endkey?
>> >>>>
>> >>>> If so, I would like to order all the views in the posted query
>> >>>> document by the result of _count from smallest to largest.
>> >>>>
>> >>>> I would then fold over the smallest result view and pull each document
>> >>>> id (*) in turn.
>> >>>>
>> >>>> With each document id I would then call each of the other views
in
>> >>>> turn with their startkey and endkey and in addition include
>> >>>> startkey_docid and endkey_docid with the docid in * above, again
>> >>>> calling _count I can test for inclusion. If the doc id is in all
views
>> >>>> then I will immediate stream this to the client.
>> >>>>
>> >>>> Am I doing something stupid, is this optimal?
>> >>>>
>> >>>
>> >>> It sounds like you are on the right track. this could be a very
>> valuable
>> >>> patch to CouchDB once you have it working.
>> >>>
>> >>>> Any help with the programmatic _count call would be great.
>> >>>>
>> >>>
>> >>> One hint: maybe the call to reduce_to_count will help.
>> >>>
>> >>> Here's an implementation of a reduce query in Erlang.
>> >>>
>> >>> http://github.com/jchris/hovercraft/blob/master/hovercraft.erl#L217
>> >>>
>> >>> Sorry I can't be more helpful. I've successfully bootstrapped this
>> stuff in
>> >>> my head before, but it always takes a couple of hours of turning my
>> brain
>> >>> into a step debugger.
>> >>>
>> >>> Good luck!
>> >>>
>> >>> Once you get deeper into the code you might have better luck getting
>> >>> responses on the dev@ list or maybe the #couchdb IRC channel on
>> freenode.
>> >>>
>> >>> Chris
>> >>>
>> >>>
>> >>>> thanks,
>> >>>>
>> >>>> Norman
>> >>>
>> >>>
>> >>
>>
>>
>

Mime
View raw message