Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of norman.barker@gmail.com
 designates 209.85.215.52 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=jYUYeOw1VZd9s4cF6stHYJXkYLutbMkts4cCTp9vDANDblfglZzRrAGfXd2FkiXd+I
         C8sdRaJAeBv5PIwDD2LJLh7q50a2Hfy7On4ui8Okty3T4dsirUA6Hufjeg/EP2a3Uo/5
         zsG45mJrI9YRZy9e2VZlwJOc7fLJUZklshsso=
MIME-Version: 1.0
In-Reply-To: <AANLkTikqu_TVh7u=p56ww4acAf3c4UTUXrg7Y3XuESXu@mail.gmail.com>
References: <AANLkTinuZG64L1c101hUf0kXux6Z47_KcEBcPoOMh4uG@mail.gmail.com>
	<AANLkTim=k9JJ11eNd=xfgpTX75zNA9dmqQ81wm87rTWo@mail.gmail.com>
	<96CEF573-64EE-44B2-AC41-2A5A312A141D@apache.org>
	<F35A86EE-AD33-4DC1-B270-1470C88D4076@apache.org>
	<AANLkTikqu_TVh7u=p56ww4acAf3c4UTUXrg7Y3XuESXu@mail.gmail.com>
Date: Mon, 16 Aug 2010 17:38:03 -0600
Message-ID: <AANLkTiko+ZG5dCgQ2vPopSGJJ6mouQtF=UYG5OdNLfQ=@mail.gmail.com>
Subject: Re: multiview on github
From: Norman Barker <norman.barker@gmail.com>
To: dev@couchdb.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi,

I have made the changes as recommended, adding a test case
multiview.js and also adding the userCtx to open the db.

I have also forked geocouch and this is available here

http://github.com/normanb/couchdb

./bootstrap
./configure
make dev
utils/run

should do it and then the simple test case is available in Futon.

The test case multiview.js takes two views which emit docs which run
from 0 .. 100, view 1 emits those documents with ids which are
multiples of 3, view 2 emits those which are multiples of 4. The
_multiview request is the intersection of view 1 and view 2 resulting
those documents whose ids are multiples of 12.

Any comments appreciated, particular concerning the following;

 1) in the module multiview, is there a quicker way to find the counts
 from startkey to endkey rather than iterating?
 2) In the module couch_query_ring is there a quicker way to test for
 inclusion rather than iterating?

Many thanks,

Norman

On Fri, Aug 6, 2010 at 10:16 AM, Norman Barker <norman.barker@gmail.com> wr=
ote:
> Chris,
>
> I will make those changes, it might be a couple of days as I am on travel=
.
>
> I will clone geocouch as a starting point and add javascript tests as
> you suggest.
>
> I am bench marking with around 10000 docs and a couple of views
> (including geocouch), the main issue with the folding over every
> document to both find the number of docs in a view slice (between
> startkey and endkey) and then again to test inclusion between views.
>
> I am interested in taking this forward and appreciate any code feedback.
>
> thanks,
>
> Norman
>
> On Thu, Aug 5, 2010 at 6:26 PM, J Chris Anderson <jchris@apache.org> wrot=
e:
>>
>> On Aug 5, 2010, at 4:32 PM, Jan Lehnardt wrote:
>>
>>> Hi Norman,
>>>
>>> I still plan to look at your code, I know the others here
>>> are fairly busy too, sorry for the review delay :)
>>>
>>
>> The code looks clean (but could use better comments about where in the f=
low each module comes into play). I don't think we can guess about performa=
nce, instead we should benchmark to make sure the ring approach is right.
>>
>> In CouchDB currently, it is possible to isolate requests against a singl=
e db. So you use the security settings to prevent access to databases, etc.=
 For this, using the userCtx and switching away from couch_db:open_int() wo=
uld make a big difference.
>>
>> This way people can query across dbs if they have read access to all of =
them.
>>
>> I think if you package this as a CouchDB fork on Github and add a few Ja=
vaScript tests, it will be really useful for some folks. I like that it has=
 geo support. Maybe we can target it for inclusion in trunk just after GeoC=
ouch goes in trunk (if Volker wants to put it in.)
>>
>> Also, for realtime hacking on this, you might find that the #couchdb IRC=
 channel on Freenode is a good place to solicit feedback. There are a lot o=
f people on there doing Geo things that would benefit from this. (They real=
ly wanna be able to intersect a Geo query with a Map Reduce query, etc.)
>>
>> Chris
>>
>>> Cheers
>>> Jan
>>> --
>>>
>>>
>>> On 5 Aug 2010, at 18:12, Norman Barker wrote:
>>>
>>>> Hi,
>>>>
>>>> is there any interest in the multiview, I have fixed (3) below, but am
>>>> still interested in approaches for (1) and (2).
>>>>
>>>> thanks,
>>>>
>>>> Norman
>>>>
>>>> On Fri, Jul 30, 2010 at 3:39 PM, Norman Barker <norman.barker@gmail.co=
m> wrote:
>>>>> Hi,
>>>>>
>>>>> a very initial version of the multiview is at
>>>>> http://github.com/normanb/couchdb-multiview for discussion.
>>>>>
>>>>> The views are intersected by using a ring of processes where each nod=
e
>>>>> in the ring represents a view as follows;
>>>>>
>>>>> % send an id from the start list to the next node in the ring, if the
>>>>> id is in adjacent node then this node sends to the next ring node ...=
.
>>>>> % if the id gets all round the ring and back to the start node then i=
t
>>>>> has intersected all queries and should be included. The nodes in the
>>>>> ring
>>>>> % should be sorted in size from small to large for this to be effecti=
ve
>>>>> %
>>>>> % In addition send the initial id list round in parallel
>>>>>
>>>>> this is implemented in the couch_query_ring module.
>>>>>
>>>>> I have a couple of questions
>>>>>
>>>>> 1) in the module multiview, is there a quicker way to find the counts
>>>>> from startkey to endkey rather than iterating?
>>>>> 2) In the module couch_query_ring is there a quicker way to test for
>>>>> inclusion rather than iterating?
>>>>> 3) Finally, if I hit this concurrently I get an exception,
>>>>>
>>>>> [error] [<0.201.0>] Uncaught error in HTTP request: {exit,
>>>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0{nopro=
c,
>>>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 {gen_=
server,call,
>>>>>
>>>>> (so ignore my previous email, I am able to trap the msg)
>>>>>
>>>>> I am going to look into (3) but if you have seen this before.
>>>>>
>>>>> I am developing on windows, but also test on linux I will work on
>>>>> getting a linux makefile, but the Makefile.win should be a start.
>>>>>
>>>>> Any help and comments appreciated.
>>>>>
>>>>> Norman
>>>>>
>>>
>>
>>
>