incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Kimber <mkim...@kana.com>
Subject RE: Am I doing something fundamentally wrong?
Date Thu, 24 May 2012 13:55:35 GMT
Robert, 

Couchdb Lists work on top of views (and look great by the way), however that brings me back
to my initial post (causes an error on this mailing list for some reason but you can find
a copy here http://mail-archives.apache.org/mod_mbox/couchdb-user/201205.mbox/%3CA7D50E04F38FD44D9D914F2ABCA592BF2E6E690685@BE259.mail.lan%3E)
:-). Namely generating a view (well a design document with views in it) on our data set takes
between 6 (simple view) and 16 hours, takes up a lot of disk space for what seems a small
amount of data and burns a CPU at 100% for the full time it runs i.e. no IO contention and
can't use multiple cores/cpus. So again am I doing something fundamentally wrong or is this
just the way Couch works and most people don't have a data set like ours so it does not take
that long to create views or does Big Couch solve the issue (although it would seem 10 big
couch nodes would still take an hour)

Looks like you work at Cloudant, so hopefully you might be able to provide some answers based
on real world experience?

Mike 



-----Original Message-----
From: Robert Newson [mailto:rnewson@apache.org] 
Sent: 24 May 2012 12:08
To: user@couchdb.apache.org
Subject: Re: Am I doing something fundamentally wrong?

Or use a list function;

http://wiki.apache.org/couchdb/Formatting_with_Show_and_List

You can use one with _all_docs and you can POST an array of ids too.

http://wiki.apache.org/couchdb/HTTP_view_API

> Since 0.9 you can also issue POST requests to views where you can send the following
JSON structure in the body:
> {"keys": ["key1", "key2", ...]}

B.

On 24 May 2012 11:58, Mike Kimber <mkimber@kana.com> wrote:
> Looking at Show documentation and running a quick test I don't think this helps as Show
has to be referenced by a doc._id or view key. If these aren't provided it returns null. This
makes sense as its for generation of a html, XML page/doc etc.
>
> So I'd have to  get a list of all doc ID's I want and then call the show function for
each and to get a filtered list I need a view.
>
> Mike
>
> -----Original Message-----
> From: Mike Kimber [mailto:mkimber@kana.com]
> Sent: 24 May 2012 10:47
> To: user@couchdb.apache.org
> Subject: RE: Am I doing something fundamentally wrong?
>
> Aurélien,
>
> Thanks for the response and apologies I didn't get a notification (e-mail) of my original
post (or the 2nd one) or your response. When I look at my original post in Google Reader is
has "An error occurred while fetching this message, sorry !", so there must be something in
the e-mail that the mailing list system does not like.
>
> In response to your original response " I'm a bit puzzled by the fact that your map functions
use the document ID". I do this because I load the data into Luciddb and this allows me to
join between tables. This is not my end game this is just a compromise due to the time it
takes to generate a view and my need to play/discover with the data.
>
> I will look at show to see if It helps, however it does not really answer my original
questions and it does not remove the more general issue that view build takes a very long
time, it only uses a single CPU and uses a bucket load of space even with compression on (no
idea why when it has a lot less data than the original)
>
> Thanks
>
> Mike
>
> -----Original Message-----
> From: Aurélien Bénel [mailto:aurelien.benel@utt.fr]
> Sent: 24 May 2012 07:40
> To: user@couchdb.apache.org
> Subject: Re: Am I doing something fundamentally wrong?
>
> Hi Mike,
>
>> Didn't seem to get there first time so having another go
>
> As I wrote in my earlier post, the use of 'map' functions in both of your examples is
overkill.
> Use 'show' functions instead.They won't require an index to be built.
>
>
> Regards,
>
> Aurélien

Mime
View raw message