couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From afshin afzali <a.afzali2...@gmail.com>
Subject Re: querying multiple views
Date Fri, 23 Jul 2010 16:44:36 GMT
Hi Norman,

As Chris said, it's a valuable patch.

Cheers,
-- afshin

On Fri, Jul 23, 2010 at 3:57 AM, Norman Barker <norman.barker@gmail.com>wrote:

> ok, I have this working in practise, though there are still many
> improvements that could be made. Sorry for the delay, 1.0
> experimenting took some of my time last week (yay for 1.0!!).
>
> Since couchdb now has an eccn number and ssl components this will take
> me a little longer to release this code, but it will go out for
> comment shortly.
>
> I did sort the views from smallest to largest (by seeing the number of
> hits in each view) and the I used the process ring example where you
> send a message in a ring, each node in the ring represents a view and
> if there is an intersection it sends a message to the next and if it
> gets back to the start then it is sent as a chunk to the user. If the
> match fails then the ring stops (actually it sends an empty message
> back to the main node).
>
> Interesting problem, but I am sure this is just a start, handling
> externals which return all their data in one blob (as they don't
> stream) is also an issue.
>
> I will work on getting the code out.
>
> Norman
>
> On Sun, Jul 11, 2010 at 11:13 AM, afshin afzali <a.afzali2003@gmail.com>
> wrote:
> > Thank You Very Much Guys.
> >
> > -- afshin
> >
> > On Sun, Jul 11, 2010 at 7:27 PM, J Chris Anderson <jchris@gmail.com>
> wrote:
> >
> >>
> >> On Jul 11, 2010, at 7:27 AM, Norman Barker wrote:
> >>
> >> > Afshin
> >> >
> >> > I have got the all clear from my work to release this as a patch, I
> >> > expect to be putting something up on github by the end of the week
> >> > (internal paperwork permitting). I am going to implement it as an
> >> > external handler so it can be used and reviewed and from there it will
> >> > be under a do what you want with it license so it can go into couchdb
> >> > if accepted.
> >> >
> >>
> >> If you're implementing in Erlang, you probably don't need to go full hog
> as
> >> an external.
> >>
> >> Because the configuration system is so modular, you are probably best
> >> adding it as a new httpd_design_handler, or httpd_db_handler.
> >>
> >> It should be easy to create a new module and link it in via the
> >> configuration file.
> >>
> >> I don't know if you plan to allow querying across databases (I'd suggest
> >> restricting the queries to a single database if you want to stay within
> >> CouchDB's security model, and have it more likely to be accepted as a
> >> patch.)
> >>
> >> We should really move this discussion to dev@ -- a lot of the
> developers
> >> only give a cursory glance at the user list, so you will get more
> valuable
> >> feedback there.
> >>
> >> Thanks for taking the time to write and release the patch!
> >>
> >> Chris
> >>
> >> > Chris, thanks for the help with the reduce function and confirming the
> >> concept!
> >> >
> >> > Norman
> >> >
> >> > On Sun, Jul 11, 2010 at 8:02 AM, afshin afzali <
> a.afzali2003@gmail.com>
> >> wrote:
> >> >> Hi Norman, Chris
> >> >>
> >> >> I just wanted to say this is the same problem we are currently facing
> >> >> with. We are implementing a Local Business Directory application on
> >> >> couchdb. Our searches need to combine several keys together to find
> >> >> right entries. To do something like that in server side, we had the
> >> >> paging mechanism problem, so we have chosen that do Norman's
> algorithm
> >> >> in client side! I'll appreciate if there will be a successful
> progress
> >> >> in this issue.
> >> >>
> >> >> BEST,
> >> >> -- afshin
> >> >>
> >> >> On 7/8/10, J Chris Anderson <jchris@apache.org> wrote:
> >> >>>
> >> >>> On Jul 8, 2010, at 10:43 AM, Norman Barker wrote:
> >> >>>
> >> >>>> Hi,
> >> >>>>
> >> >>>> I have been thinking about how to query multiple views at one
time.
> >> >>>>
> >> >>>> I have an erlang handler in couchdb that takes a http post
> containing
> >> >>>> N view queries, each query contains a startkey and an endkey,
I
> then
> >> >>>> open up each view in parallel (using pmap) and accumulate the
doc
> ids,
> >> >>>> then I use the erlang sets module to get the unique values.
All
> good
> >> >>>> and looks pretty (and works), though it doesn't scale since
I am
> >> >>>> holding all the results on the server (potential memory overload!)
> >> >>>> whereas I would like to stream the results to the client one
by
> one.
> >> >>>>
> >> >>>> I am thinking of doing the following but have some questions;
> >> >>>>
> >> >>>> My first question is when I do
> >> >>>>
> >> >>>> couch_view:fold(View, FoldlFun, FoldAccInit,
> >> >>>> couch_httpd_view:make_key_options(Args)),
> >> >>>>
> >> >>>> is there a way to call the _count reduce function in code to
find
> the
> >> >>>> number of rows in the slice between startkey and endkey?
> >> >>>>
> >> >>>> If so, I would like to order all the views in the posted query
> >> >>>> document by the result of _count from smallest to largest.
> >> >>>>
> >> >>>> I would then fold over the smallest result view and pull each
> document
> >> >>>> id (*) in turn.
> >> >>>>
> >> >>>> With each document id I would then call each of the other views
in
> >> >>>> turn with their startkey and endkey and in addition include
> >> >>>> startkey_docid and endkey_docid with the docid in * above,
again
> >> >>>> calling _count I can test for inclusion. If the doc id is in
all
> views
> >> >>>> then I will immediate stream this to the client.
> >> >>>>
> >> >>>> Am I doing something stupid, is this optimal?
> >> >>>>
> >> >>>
> >> >>> It sounds like you are on the right track. this could be a very
> >> valuable
> >> >>> patch to CouchDB once you have it working.
> >> >>>
> >> >>>> Any help with the programmatic _count call would be great.
> >> >>>>
> >> >>>
> >> >>> One hint: maybe the call to reduce_to_count will help.
> >> >>>
> >> >>> Here's an implementation of a reduce query in Erlang.
> >> >>>
> >> >>> http://github.com/jchris/hovercraft/blob/master/hovercraft.erl#L217
> >> >>>
> >> >>> Sorry I can't be more helpful. I've successfully bootstrapped this
> >> stuff in
> >> >>> my head before, but it always takes a couple of hours of turning
my
> >> brain
> >> >>> into a step debugger.
> >> >>>
> >> >>> Good luck!
> >> >>>
> >> >>> Once you get deeper into the code you might have better luck getting
> >> >>> responses on the dev@ list or maybe the #couchdb IRC channel on
> >> freenode.
> >> >>>
> >> >>> Chris
> >> >>>
> >> >>>
> >> >>>> thanks,
> >> >>>>
> >> >>>> Norman
> >> >>>
> >> >>>
> >> >>
> >>
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message