Return-Path: Delivered-To: apmail-couchdb-dev-archive@www.apache.org Received: (qmail 30927 invoked from network); 18 Sep 2010 03:32:36 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 18 Sep 2010 03:32:36 -0000 Received: (qmail 22428 invoked by uid 500); 18 Sep 2010 03:32:35 -0000 Delivered-To: apmail-couchdb-dev-archive@couchdb.apache.org Received: (qmail 22016 invoked by uid 500); 18 Sep 2010 03:32:33 -0000 Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@couchdb.apache.org Delivered-To: mailing list dev@couchdb.apache.org Received: (qmail 22003 invoked by uid 99); 18 Sep 2010 03:32:32 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Sep 2010 03:32:32 +0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of norman.barker@gmail.com designates 74.125.82.180 as permitted sender) Received: from [74.125.82.180] (HELO mail-wy0-f180.google.com) (74.125.82.180) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Sep 2010 03:32:28 +0000 Received: by wyb40 with SMTP id 40so4118390wyb.11 for ; Fri, 17 Sep 2010 20:32:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=OtS5a57uRnPoylwfsASZoOGG7Nq2JOZ6x/jZLHQ8C2Q=; b=ObGQn3QSuNL3LNrhe+geIFJKI+J3myRtpSXIqXxspGS5oGrjhWoVhurSKDEjSJ18NN kAQ6FicP0YHtEEIO5apd1crj89BwBaKgXnQ1W4ePLyRlrD8O0xZxJeSpADZtWG22AP/N eFJ6GLMc0aLa7aeQzBIoefHgMvlxPZRI/VoeU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=hezDsBTq3UmjzI46aKENU07R9x4yP5ffFXL8/cMiHzTWctrlkNxRCfGbMrw5gVIuqh nNP5h3Mo98/pIJGDXd8oaHNUPg1A2QKpzWhY3OyQ8uBYDomct7n5v2Eo/IK21hR6RlU0 jW3bhjDauyQmOMAjeWegyYK7HVvSu3o7r9RBY= MIME-Version: 1.0 Received: by 10.216.17.207 with SMTP id j57mr1532576wej.68.1284780727114; Fri, 17 Sep 2010 20:32:07 -0700 (PDT) Received: by 10.216.69.212 with HTTP; Fri, 17 Sep 2010 20:32:07 -0700 (PDT) In-Reply-To: References: <96CEF573-64EE-44B2-AC41-2A5A312A141D@apache.org> <4835209A-F0C3-493E-8ED9-E58FEC4B1C55@apache.org> <4C6B3C3B.8020607@gmail.com> Date: Fri, 17 Sep 2010 21:32:07 -0600 Message-ID: Subject: Re: multiview on github From: Norman Barker To: dev@couchdb.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Chris, James thanks for bumping this, we are using this internally at 'scale' (million+ keys). I want this to work for couchdb as we want to give back for such a great product and support this going forward, so any suggestions welcomed and we will test and add them to the local github account with the aim of getting this into trunk. Norman On Fri, Sep 17, 2010 at 7:00 PM, James Hayton w= rote: > I want to use it! =A0I just haven't gotten around to it. =A0I was going t= o try > and test it out this weekend and if I am able, I will certainly report ba= ck > what I find. > > James > > On Fri, Sep 17, 2010 at 5:55 PM, Chris Anderson wrote= : > >> On Mon, Aug 30, 2010 at 10:58 AM, Norman Barker >> wrote: >> > Bob, >> > >> > I can and have been testing the multiview at this scale, it is ok >> > (fast enough), but I think being able to test inclusion of a document >> > id in a view without having to loop would be a considerable speed >> > improvement. If you have any ideas let me know. >> > >> >> I just want to bump this thread, as I think this is a useful feature. >> I don't expect to be able to test it in the coming weeks, but if I did >> I would. Is anyone besides Norman using this? Has anyone used it at >> scale? >> >> Cheers, >> Chris >> >> > thanks, >> > >> > Norman >> > >> > On Mon, Aug 30, 2010 at 10:49 AM, Robert Newson >> wrote: >> >> I'm sorry, I've had no time to play with this at scale. >> >> >> >> On Mon, Aug 30, 2010 at 5:35 PM, Norman Barker >> wrote: >> >>> Hi, >> >>> >> >>> are there any more comments on this, if not can you describe the >> >>> process (in particular how to obtain a wiki and jira account for >> >>> couchdb which I have been unable to do) and I will start documenting >> >>> this so we can put this into the trunk. >> >>> >> >>> Bob, were you able to do any more testing with large views, are ther= e >> >>> any suggestions on how to speed up the document id inclusion test as >> >>> described below? >> >>> >> >>> thanks, >> >>> >> >>> Norman >> >>> >> >>> On Mon, Aug 23, 2010 at 9:22 AM, Norman Barker < >> norman.barker@gmail.com> wrote: >> >>>> Bob, >> >>>> >> >>>> thanks for the feedback and for taking a look at the code. Guidelin= es >> >>>> on when to use a supervisor within couchdb with a gen_server would = be >> >>>> appreciated, currently I have a supervisor and a gen_server, but if >> >>>> couchdb has a supervision process I could remove that layer. >> >>>> >> >>>> I think plugins is a great idea, however intersection of views is s= uch >> >>>> as common request, perhaps there needs to plugin system and if a >> >>>> plugin is rated enough it goes into trunk as a core feature. >> >>>> >> >>>> the four (or slightly more) summary is here >> >>>> >> >>>> >> http://github.com/normanb/couchdb/raw/trunk/src/couchdb/couch_query_ring= .erl >> >>>> >> >>>> % >> >>>> % send an id from the start list to the next node in the ring, if t= he >> >>>> id is in adjacent node then the this node sends to the next ring no= de >> >>>> .... >> >>>> % if the id gets all round the ring and back to the start node then= is >> >>>> has intersected all queries and should be included. The nodes in th= e >> >>>> ring >> >>>> % should be sorted in size from small to large for this to be >> effective >> >>>> % >> >>>> % In addition send the initial id list round in parallel >> >>>> >> >>>> it really needs some eyes from the core couchdb coders to see how t= o >> >>>> speed up the inclusion testing, looping is bad even if it is done i= n >> >>>> parallel. >> >>>> >> >>>> Multiview is usable, I am using it with some pretty big mega-views = (as >> >>>> per the raindrop) model, I am also available to add features to thi= s >> >>>> as this is core part of our work and we want to give it to couch as= a >> >>>> contribution. >> >>>> >> >>>> thanks, >> >>>> >> >>>> Norman >> >>>> >> >>>> On Mon, Aug 23, 2010 at 5:05 AM, Robert Dionne >> >>>> wrote: >> >>>>> Hi Norman, >> >>>>> >> >>>>> =A0I took a peek at multiview. I haven't followed this too closely= on >> the mailing list but this is *view intersection*? Is there a 5 line summ= ary >> of what this does somewhere? >> >>>>> >> >>>>> =A0I'm curious as to why the daemon needs to be a supervisor, most= if >> not all of the other daemons are gen_servers. OTP allows this but I thin= k >> this is a good area where some CouchDB guidelines on plugins would apply= . >> >>>>> >> >>>>> =A0It strikes me that views, the use of map/reduce, etc. are one o= f the >> trickier aspects of using CouchDB, particularly for new users coming fro= m >> the SQL world. People are also reporting issues with performance of view= s, I >> guess often because reduce functions go out of control. >> >>>>> >> >>>>> =A0I think the project would be better served if features like thi= s >> were available as plugins. I would put GeoCouch in the same category. It= s >> very neat and timely (given everyone wants to know where everyone else i= s >> using their telephone but without talking other than asynchronously), bu= t a >> server plugin architecture that would allow this to be done cleanly shou= ld >> come first. >> >>>>> >> >>>>> =A0This is just my opinion. I'd love to see some of the project >> founders and committers weigh in on this and set some direction. >> >>>>> >> >>>>> Best regards, >> >>>>> >> >>>>> Bob >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> On Aug 22, 2010, at 5:45 PM, Norman Barker wrote: >> >>>>> >> >>>>>> I would like to take this multiview code and have it added to tru= nk >> if >> >>>>>> possible, what are the next steps? >> >>>>>> >> >>>>>> thanks, >> >>>>>> >> >>>>>> Norman >> >>>>>> >> >>>>>> On Wed, Aug 18, 2010 at 11:44 AM, Norman Barker < >> norman.barker@gmail.com> wrote: >> >>>>>>> I have made >> >>>>>>> >> >>>>>>> http://github.com/normanb/couchdb >> >>>>>>> >> >>>>>>> which is a fork of the latest couchdb trunk with the multiview c= ode >> >>>>>>> and tests added. >> >>>>>>> >> >>>>>>> If geocouch is available then it can still be used. >> >>>>>>> >> >>>>>>> There are a couple of questions about the multiview on the user >> /dev >> >>>>>>> list so I will be adding some more test cases during today. >> >>>>>>> >> >>>>>>> thanks, >> >>>>>>> >> >>>>>>> Norman >> >>>>>>> >> >>>>>>> On Tue, Aug 17, 2010 at 9:23 PM, Norman Barker < >> norman.barker@gmail.com> wrote: >> >>>>>>>> this is possible, I forked geocouch since I use it, but I have >> already >> >>>>>>>> separated the geocouch dependencies from the trunk. >> >>>>>>>> >> >>>>>>>> I can do this tomorrow, certainly be interested in any feedback= . >> >>>>>>>> >> >>>>>>>> thanks, >> >>>>>>>> >> >>>>>>>> Norman >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> On Tue, Aug 17, 2010 at 7:49 PM, Volker Mische < >> volker.mische@gmail.com> wrote: >> >>>>>>>>> On 08/18/2010 03:26 AM, J Chris Anderson wrote: >> >>>>>>>>>> >> >>>>>>>>>> On Aug 16, 2010, at 4:38 PM, Norman Barker wrote: >> >>>>>>>>>> >> >>>>>>>>>>> Hi, >> >>>>>>>>>>> >> >>>>>>>>>>> I have made the changes as recommended, adding a test case >> >>>>>>>>>>> multiview.js and also adding the userCtx to open the db. >> >>>>>>>>>>> >> >>>>>>>>>>> I have also forked geocouch and this is available here >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> this patch seems important (especially as people are already >> asking for >> >>>>>>>>>> help using it on user@) >> >>>>>>>>>> >> >>>>>>>>>> to get it committed, it either must remove the dependency on >> GeoCouch, or >> >>>>>>>>>> become part of CouchDB when (and if) GeoCouch becomes part of >> CouchDB. >> >>>>>>>>>> >> >>>>>>>>>> Is it possible / useful to make a version that doesn't use >> GeoCouch? And >> >>>>>>>>>> then to make the GeoCouch capabilities part GeoCouch for now? >> >>>>>>>>>> >> >>>>>>>>>> Chris >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> Hi Norman, >> >>>>>>>>> >> >>>>>>>>> if the patch is ready for trunk, I'd be happy to move the >> GeoCouch bits to >> >>>>>>>>> GeoCouch itself (as GeoCouch isn't ready for trunk yet). >> >>>>>>>>> >> >>>>>>>>> Lately I haven't been that responsive when it comes to GeoCouc= h, >> but that >> >>>>>>>>> will change (in about a month) after holidays and FOSS4G. >> >>>>>>>>> >> >>>>>>>>> Cheers, >> >>>>>>>>> =A0Volker >> >>>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>> >> >>>>> >> >>>> >> >>> >> >> >> > >> >> >> >> -- >> Chris Anderson >> http://jchrisa.net >> http://couch.io >> >