incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Anderson" <jch...@grabb.it>
Subject Re: Is it possible to evaluate a view on a 20.000 documents database?
Date Thu, 31 Jul 2008 22:14:30 GMT
If your view is complex, and you have many (100k+) records (and the
emitted row size is large) views could take hours to generate on a
Core Duo MacBook. Let them generate overnight, and in the morning the
queries will be very fast.

On Thu, Jul 31, 2008 at 2:44 PM, Ed Finkler <funkatron@gmail.com> wrote:
> I have been working with a very similar problem, actually. A large set of
> records (40k+), building views from scratch.
>
> My experience was that I just needed to let couchdb build the view. It can
> take several minutes, and the CPU usage will be high. You should see both
> the beam and couchjs processes working while the view is building. If you're
> accessing a view via Futon, it's likely the browser will time-out the
> request before the build is finished. The build process *will* continue on
> the server side, though. If you let the build finish, the next time you
> query the view, it will return the data immediately.
>
> To mitigate this problem, I'm now updating the view every time I do an
> insert (I bulk-add 20 records per minute). This only requires that the new
> data be added to the view, so building at this point is a short process.
>
> (big thanks to the folks on #couchdb for helping me with this problem!)
>
> --
> Ed Finkler
> http://funkatron.com
> AIM: funka7ron
> ICQ: 3922133
> Skype: funka7ron
>
>
> On Jul 31, 2008, at 5:10 PM, Demetrius Nunes wrote:
>
>> Hi there,
>>
>> I was having a great time playing aroung with CouchDB. It seems like a
>> perfect fit for a future system that we'll be building fairly soon.
>>
>> But then, I've just created a CouchDB database, importing 20.000 records
>> from an old relational database into it.
>>
>> When I go into Futon, I can see the database is there, with 20.899
>> documents
>> and 125.2 MB in size.
>>
>> Clicking on it, I can navigate thru the "All Documents" pretty quickly (10
>> documents per page).
>>
>> The problem is when I try to create a custom view. Just as I enter the
>> custom view page in Futon, the server hangs and locks up my CPU at 90%
>> usage. I waited several minutes for it to cool off but the process was
>> still
>> there and I had no response at all.
>>
>> I then tried to create a view programatically, using REST/JSON and I get
>> the
>> same result.
>>
>> I am running CouchDB 0.8.0 on Ubuntu 8.0.4.
>>
>> Is CouchDB not ready for a dataset of this size yet?
>>
>> Thanks and best regards,
>> Dema
>>
>> --
>> ____________________________
>> http://www.demetriusnunes.com
>
>



-- 
Chris Anderson
http://jchris.mfdz.com

Mime
View raw message