incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Kimber <mkim...@kana.com>
Subject RE: Am I doing something fundamentally wrong?
Date Fri, 25 May 2012 09:45:46 GMT
I tried it and to be quiet frank I'm not surprised! Its reduced it from 5.6GB to 4.5GB which
is less than the automatic compaction threshold for views set on our install, which is why
that has not kicked in. Either way for what is effectively storing 3 name value pair's and
a Key this is an absolutely massive amount of space (equivalent takes 400MB on a Column Orientated
Database 258 Million rows) and as Couchdb 1.2 has compression on by default for views and
databases this is compressed, so what the heck is in it???

I know disk space is regarded as being cheap and cloudant runs on cloud infrastructure so
CPU can be brought in on demand, but the amount of disk space and length of time it takes
burning CPU to rebuild the views has to be increase your operational costs significantly or
does the Cloudant BigCouch version have some secret source in it that optimises this? This
is why Facebook moved to RCFILE for Hadoop/Hive. 

This is my only bug bear with Couchdb and I'd like to see optimisation in this area on the
Couchdb Roadmap (assuming this is normal couchdb behaviour and my install is not up the left
in some way), but it does seem that I'm the only one on this mailing list who sees this as
an issue. Either everyone else does not have lots of data or their employers/VC's are happy
to throw hardware at the problem or they like twiddling their thumbs as their views build.


Thanks

Mike 

-----Original Message-----
From: Robert Newson [mailto:rnewson@apache.org] 
Sent: 24 May 2012 15:22
To: user@couchdb.apache.org
Subject: Re: Am I doing something fundamentally wrong?

Databases (and views) need compacting even if you never update or
delete a document. Try it, you might be surprised.

B.

On 24 May 2012 15:19, Sean Copenhaver <sean.copenhaver@gmail.com> wrote:
> I believe multiple design documents will build views concurrently but one
> design document is basically done sequentially by the change sequence...
> not positive.
>
> So you could try splitting out your views into multiple design documents
> and hit them to see if that helps spread out the CPU usage. I want to say a
> lot of the CPU usage is the serialization process that is happening
> communicating from CouchDB's core to the view engine process.
>
> Anyway with the list you specify any view and all_docs is a view with all
> documents in a database. So if you know the ids you want to work with you
> can doe a normal view query with a list function.
> http://wiki.apache.org/couchdb/HTTP_Document_API#all_docs
>
> That's what Robert was trying to get at.
>
> On Thu, May 24, 2012 at 9:55 AM, Mike Kimber <mkimber@kana.com> wrote:
>
>> Robert,
>>
>> Couchdb Lists work on top of views (and look great by the way), however
>> that brings me back to my initial post (causes an error on this mailing
>> list for some reason but you can find a copy here
>> http://mail-archives.apache.org/mod_mbox/couchdb-user/201205.mbox/%3CA7D50E04F38FD44D9D914F2ABCA592BF2E6E690685@BE259.mail.lan%3E)
>> :-). Namely generating a view (well a design document with views in it) on
>> our data set takes between 6 (simple view) and 16 hours, takes up a lot of
>> disk space for what seems a small amount of data and burns a CPU at 100%
>> for the full time it runs i.e. no IO contention and can't use multiple
>> cores/cpus. So again am I doing something fundamentally wrong or is this
>> just the way Couch works and most people don't have a data set like ours so
>> it does not take that long to create views or does Big Couch solve the
>> issue (although it would seem 10 big couch nodes would still take an hour)
>>
>> Looks like you work at Cloudant, so hopefully you might be able to provide
>> some answers based on real world experience?
>>
>> Mike
>>
>>
>>
>> -----Original Message-----
>> From: Robert Newson [mailto:rnewson@apache.org]
>> Sent: 24 May 2012 12:08
>> To: user@couchdb.apache.org
>> Subject: Re: Am I doing something fundamentally wrong?
>>
>> Or use a list function;
>>
>> http://wiki.apache.org/couchdb/Formatting_with_Show_and_List
>>
>> You can use one with _all_docs and you can POST an array of ids too.
>>
>> http://wiki.apache.org/couchdb/HTTP_view_API
>>
>> > Since 0.9 you can also issue POST requests to views where you can send
>> the following JSON structure in the body:
>> > {"keys": ["key1", "key2", ...]}
>>
>> B.
>>
>> On 24 May 2012 11:58, Mike Kimber <mkimber@kana.com> wrote:
>> > Looking at Show documentation and running a quick test I don't think
>> this helps as Show has to be referenced by a doc._id or view key. If these
>> aren't provided it returns null. This makes sense as its for generation of
>> a html, XML page/doc etc.
>> >
>> > So I'd have to  get a list of all doc ID's I want and then call the show
>> function for each and to get a filtered list I need a view.
>> >
>> > Mike
>> >
>> > -----Original Message-----
>> > From: Mike Kimber [mailto:mkimber@kana.com]
>> > Sent: 24 May 2012 10:47
>> > To: user@couchdb.apache.org
>> > Subject: RE: Am I doing something fundamentally wrong?
>> >
>> > Aurélien,
>> >
>> > Thanks for the response and apologies I didn't get a notification
>> (e-mail) of my original post (or the 2nd one) or your response. When I look
>> at my original post in Google Reader is has "An error occurred while
>> fetching this message, sorry !", so there must be something in the e-mail
>> that the mailing list system does not like.
>> >
>> > In response to your original response " I'm a bit puzzled by the fact
>> that your map functions use the document ID". I do this because I load the
>> data into Luciddb and this allows me to join between tables. This is not my
>> end game this is just a compromise due to the time it takes to generate a
>> view and my need to play/discover with the data.
>> >
>> > I will look at show to see if It helps, however it does not really
>> answer my original questions and it does not remove the more general issue
>> that view build takes a very long time, it only uses a single CPU and uses
>> a bucket load of space even with compression on (no idea why when it has a
>> lot less data than the original)
>> >
>> > Thanks
>> >
>> > Mike
>> >
>> > -----Original Message-----
>> > From: Aurélien Bénel [mailto:aurelien.benel@utt.fr]
>> > Sent: 24 May 2012 07:40
>> > To: user@couchdb.apache.org
>> > Subject: Re: Am I doing something fundamentally wrong?
>> >
>> > Hi Mike,
>> >
>> >> Didn't seem to get there first time so having another go
>> >
>> > As I wrote in my earlier post, the use of 'map' functions in both of
>> your examples is overkill.
>> > Use 'show' functions instead.They won't require an index to be built.
>> >
>> >
>> > Regards,
>> >
>> > Aurélien
>>
>
>
>
> --
> "The limits of language are the limits of one's world. " - Ludwig von
> Wittgenstein
>
> "Water is fluid, soft and yielding. But water will wear away rock, which is
> rigid and cannot yield. As a rule, whatever is fluid, soft and yielding
> will overcome whatever is rigid and hard. This is another paradox: what is
> soft is strong." - Lao-Tzu

Mime
View raw message