From user-return-20948-apmail-couchdb-user-archive=couchdb.apache.org@couchdb.apache.org Thu May 24 14:21:50 2012 Return-Path: X-Original-To: apmail-couchdb-user-archive@www.apache.org Delivered-To: apmail-couchdb-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DC830C724 for ; Thu, 24 May 2012 14:21:50 +0000 (UTC) Received: (qmail 51520 invoked by uid 500); 24 May 2012 14:21:49 -0000 Delivered-To: apmail-couchdb-user-archive@couchdb.apache.org Received: (qmail 51473 invoked by uid 500); 24 May 2012 14:21:49 -0000 Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@couchdb.apache.org Delivered-To: mailing list user@couchdb.apache.org Received: (qmail 51464 invoked by uid 99); 24 May 2012 14:21:49 -0000 Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 May 2012 14:21:49 +0000 Received: from localhost (HELO mail-yw0-f52.google.com) (127.0.0.1) (smtp-auth username rnewson, mechanism plain) by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 May 2012 14:21:48 +0000 Received: by yhpp61 with SMTP id p61so9484034yhp.11 for ; Thu, 24 May 2012 07:21:47 -0700 (PDT) MIME-Version: 1.0 Received: by 10.50.181.232 with SMTP id dz8mr16831840igc.72.1337869307522; Thu, 24 May 2012 07:21:47 -0700 (PDT) Received: by 10.42.110.76 with HTTP; Thu, 24 May 2012 07:21:47 -0700 (PDT) In-Reply-To: References: <08E52809-C962-4E9C-AFB8-397EA201580E@utt.fr> Date: Thu, 24 May 2012 15:21:47 +0100 Message-ID: Subject: Re: Am I doing something fundamentally wrong? From: Robert Newson To: user@couchdb.apache.org Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Databases (and views) need compacting even if you never update or delete a document. Try it, you might be surprised. B. On 24 May 2012 15:19, Sean Copenhaver wrote: > I believe multiple design documents will build views concurrently but one > design document is basically done sequentially by the change sequence... > not positive. > > So you could try splitting out your views into multiple design documents > and hit them to see if that helps spread out the CPU usage. I want to say= a > lot of the CPU usage is the serialization process that is happening > communicating from CouchDB's core to the view engine process. > > Anyway with the list you specify any view and all_docs is a view with all > documents in a database. So if you know the ids you want to work with you > can doe a normal view query with a list function. > http://wiki.apache.org/couchdb/HTTP_Document_API#all_docs > > That's what Robert was trying to get at. > > On Thu, May 24, 2012 at 9:55 AM, Mike Kimber wrote: > >> Robert, >> >> Couchdb Lists work on top of views (and look great by the way), however >> that brings me back to my initial post (causes an error on this mailing >> list for some reason but you can find a copy here >> http://mail-archives.apache.org/mod_mbox/couchdb-user/201205.mbox/%3CA7D= 50E04F38FD44D9D914F2ABCA592BF2E6E690685@BE259.mail.lan%3E) >> :-). Namely generating a view (well a design document with views in it) = on >> our data set takes between 6 (simple view) and 16 hours, takes up a lot = of >> disk space for what seems a small amount of data and burns a CPU at 100% >> for the full time it runs i.e. no IO contention and can't use multiple >> cores/cpus. So again am I doing something fundamentally wrong or is this >> just the way Couch works and most people don't have a data set like ours= so >> it does not take that long to create views or does Big Couch solve the >> issue (although it would seem 10 big couch nodes would still take an hou= r) >> >> Looks like you work at Cloudant, so hopefully you might be able to provi= de >> some answers based on real world experience? >> >> Mike >> >> >> >> -----Original Message----- >> From: Robert Newson [mailto:rnewson@apache.org] >> Sent: 24 May 2012 12:08 >> To: user@couchdb.apache.org >> Subject: Re: Am I doing something fundamentally wrong? >> >> Or use a list function; >> >> http://wiki.apache.org/couchdb/Formatting_with_Show_and_List >> >> You can use one with _all_docs and you can POST an array of ids too. >> >> http://wiki.apache.org/couchdb/HTTP_view_API >> >> > Since 0.9 you can also issue POST requests to views where you can send >> the following JSON structure in the body: >> > {"keys": ["key1", "key2", ...]} >> >> B. >> >> On 24 May 2012 11:58, Mike Kimber wrote: >> > Looking at Show documentation and running a quick test I don't think >> this helps as Show has to be referenced by a doc._id or view key. If the= se >> aren't provided it returns null. This makes sense as its for generation = of >> a html, XML page/doc etc. >> > >> > So I'd have to =A0get a list of all doc ID's I want and then call the = show >> function for each and to get a filtered list I need a view. >> > >> > Mike >> > >> > -----Original Message----- >> > From: Mike Kimber [mailto:mkimber@kana.com] >> > Sent: 24 May 2012 10:47 >> > To: user@couchdb.apache.org >> > Subject: RE: Am I doing something fundamentally wrong? >> > >> > Aur=E9lien, >> > >> > Thanks for the response and apologies I didn't get a notification >> (e-mail) of my original post (or the 2nd one) or your response. When I l= ook >> at my original post in Google Reader is has "An error occurred while >> fetching this message, sorry !", so there must be something in the e-mai= l >> that the mailing list system does not like. >> > >> > In response to your original response " I'm a bit puzzled by the fact >> that your map functions use the document ID". I do this because I load t= he >> data into Luciddb and this allows me to join between tables. This is not= my >> end game this is just a compromise due to the time it takes to generate = a >> view and my need to play/discover with the data. >> > >> > I will look at show to see if It helps, however it does not really >> answer my original questions and it does not remove the more general iss= ue >> that view build takes a very long time, it only uses a single CPU and us= es >> a bucket load of space even with compression on (no idea why when it has= a >> lot less data than the original) >> > >> > Thanks >> > >> > Mike >> > >> > -----Original Message----- >> > From: Aur=E9lien B=E9nel [mailto:aurelien.benel@utt.fr] >> > Sent: 24 May 2012 07:40 >> > To: user@couchdb.apache.org >> > Subject: Re: Am I doing something fundamentally wrong? >> > >> > Hi Mike, >> > >> >> Didn't seem to get there first time so having another go >> > >> > As I wrote in my earlier post, the use of 'map' functions in both of >> your examples is overkill. >> > Use 'show' functions instead.They won't require an index to be built. >> > >> > >> > Regards, >> > >> > Aur=E9lien >> > > > > -- > =93The limits of language are the limits of one's world. =93 - Ludwig von > Wittgenstein > > "Water is fluid, soft and yielding. But water will wear away rock, which = is > rigid and cannot yield. As a rule, whatever is fluid, soft and yielding > will overcome whatever is rigid and hard. This is another paradox: what i= s > soft is strong." - Lao-Tzu