Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Sender: J Chris Anderson <jchris@couch.io>
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1081)
Subject: Re: multiview on github
From: J Chris Anderson <jchris@apache.org>
In-Reply-To: <AANLkTiko+ZG5dCgQ2vPopSGJJ6mouQtF=UYG5OdNLfQ=@mail.gmail.com>
Date: Tue, 17 Aug 2010 18:26:21 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <4835209A-F0C3-493E-8ED9-E58FEC4B1C55@apache.org>
References: <AANLkTinuZG64L1c101hUf0kXux6Z47_KcEBcPoOMh4uG@mail.gmail.com>
 <AANLkTim=k9JJ11eNd=xfgpTX75zNA9dmqQ81wm87rTWo@mail.gmail.com>
 <96CEF573-64EE-44B2-AC41-2A5A312A141D@apache.org>
 <F35A86EE-AD33-4DC1-B270-1470C88D4076@apache.org>
 <AANLkTikqu_TVh7u=p56ww4acAf3c4UTUXrg7Y3XuESXu@mail.gmail.com>
 <AANLkTiko+ZG5dCgQ2vPopSGJJ6mouQtF=UYG5OdNLfQ=@mail.gmail.com>
To: dev@couchdb.apache.org


On Aug 16, 2010, at 4:38 PM, Norman Barker wrote:

> Hi,
>=20
> I have made the changes as recommended, adding a test case
> multiview.js and also adding the userCtx to open the db.
>=20
> I have also forked geocouch and this is available here
>=20

this patch seems important (especially as people are already asking for =
help using it on user@)

to get it committed, it either must remove the dependency on GeoCouch, =
or become part of CouchDB when (and if) GeoCouch becomes part of =
CouchDB.

Is it possible / useful to make a version that doesn't use GeoCouch? And =
then to make the GeoCouch capabilities part GeoCouch for now?

Chris

> http://github.com/normanb/couchdb
>=20
> ./bootstrap
> ./configure
> make dev
> utils/run
>=20
> should do it and then the simple test case is available in Futon.
>=20
> The test case multiview.js takes two views which emit docs which run
> from 0 .. 100, view 1 emits those documents with ids which are
> multiples of 3, view 2 emits those which are multiples of 4. The
> _multiview request is the intersection of view 1 and view 2 resulting
> those documents whose ids are multiples of 12.
>=20
> Any comments appreciated, particular concerning the following;
>=20
> 1) in the module multiview, is there a quicker way to find the counts
> from startkey to endkey rather than iterating?
> 2) In the module couch_query_ring is there a quicker way to test for
> inclusion rather than iterating?
>=20
> Many thanks,
>=20
> Norman
>=20
> On Fri, Aug 6, 2010 at 10:16 AM, Norman Barker =
<norman.barker@gmail.com> wrote:
>> Chris,
>>=20
>> I will make those changes, it might be a couple of days as I am on =
travel.
>>=20
>> I will clone geocouch as a starting point and add javascript tests as
>> you suggest.
>>=20
>> I am bench marking with around 10000 docs and a couple of views
>> (including geocouch), the main issue with the folding over every
>> document to both find the number of docs in a view slice (between
>> startkey and endkey) and then again to test inclusion between views.
>>=20
>> I am interested in taking this forward and appreciate any code =
feedback.
>>=20
>> thanks,
>>=20
>> Norman
>>=20
>> On Thu, Aug 5, 2010 at 6:26 PM, J Chris Anderson <jchris@apache.org> =
wrote:
>>>=20
>>> On Aug 5, 2010, at 4:32 PM, Jan Lehnardt wrote:
>>>=20
>>>> Hi Norman,
>>>>=20
>>>> I still plan to look at your code, I know the others here
>>>> are fairly busy too, sorry for the review delay :)
>>>>=20
>>>=20
>>> The code looks clean (but could use better comments about where in =
the flow each module comes into play). I don't think we can guess about =
performance, instead we should benchmark to make sure the ring approach =
is right.
>>>=20
>>> In CouchDB currently, it is possible to isolate requests against a =
single db. So you use the security settings to prevent access to =
databases, etc. For this, using the userCtx and switching away from =
couch_db:open_int() would make a big difference.
>>>=20
>>> This way people can query across dbs if they have read access to all =
of them.
>>>=20
>>> I think if you package this as a CouchDB fork on Github and add a =
few JavaScript tests, it will be really useful for some folks. I like =
that it has geo support. Maybe we can target it for inclusion in trunk =
just after GeoCouch goes in trunk (if Volker wants to put it in.)
>>>=20
>>> Also, for realtime hacking on this, you might find that the #couchdb =
IRC channel on Freenode is a good place to solicit feedback. There are a =
lot of people on there doing Geo things that would benefit from this. =
(They really wanna be able to intersect a Geo query with a Map Reduce =
query, etc.)
>>>=20
>>> Chris
>>>=20
>>>> Cheers
>>>> Jan
>>>> --
>>>>=20
>>>>=20
>>>> On 5 Aug 2010, at 18:12, Norman Barker wrote:
>>>>=20
>>>>> Hi,
>>>>>=20
>>>>> is there any interest in the multiview, I have fixed (3) below, =
but am
>>>>> still interested in approaches for (1) and (2).
>>>>>=20
>>>>> thanks,
>>>>>=20
>>>>> Norman
>>>>>=20
>>>>> On Fri, Jul 30, 2010 at 3:39 PM, Norman Barker =
<norman.barker@gmail.com> wrote:
>>>>>> Hi,
>>>>>>=20
>>>>>> a very initial version of the multiview is at
>>>>>> http://github.com/normanb/couchdb-multiview for discussion.
>>>>>>=20
>>>>>> The views are intersected by using a ring of processes where each =
node
>>>>>> in the ring represents a view as follows;
>>>>>>=20
>>>>>> % send an id from the start list to the next node in the ring, if =
the
>>>>>> id is in adjacent node then this node sends to the next ring node =
....
>>>>>> % if the id gets all round the ring and back to the start node =
then it
>>>>>> has intersected all queries and should be included. The nodes in =
the
>>>>>> ring
>>>>>> % should be sorted in size from small to large for this to be =
effective
>>>>>> %
>>>>>> % In addition send the initial id list round in parallel
>>>>>>=20
>>>>>> this is implemented in the couch_query_ring module.
>>>>>>=20
>>>>>> I have a couple of questions
>>>>>>=20
>>>>>> 1) in the module multiview, is there a quicker way to find the =
counts
>>>>>> from startkey to endkey rather than iterating?
>>>>>> 2) In the module couch_query_ring is there a quicker way to test =
for
>>>>>> inclusion rather than iterating?
>>>>>> 3) Finally, if I hit this concurrently I get an exception,
>>>>>>=20
>>>>>> [error] [<0.201.0>] Uncaught error in HTTP request: {exit,
>>>>>>                                {noproc,
>>>>>>                                 {gen_server,call,
>>>>>>=20
>>>>>> (so ignore my previous email, I am able to trap the msg)
>>>>>>=20
>>>>>> I am going to look into (3) but if you have seen this before.
>>>>>>=20
>>>>>> I am developing on windows, but also test on linux I will work on
>>>>>> getting a linux makefile, but the Makefile.win should be a start.
>>>>>>=20
>>>>>> Any help and comments appreciated.
>>>>>>=20
>>>>>> Norman
>>>>>>=20
>>>>=20
>>>=20
>>>=20
>>=20