couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ed Finkler <funkat...@gmail.com>
Subject Re: Is it possible to evaluate a view on a 20.000 documents database?
Date Thu, 31 Jul 2008 21:44:45 GMT
I have been working with a very similar problem, actually. A large set  
of records (40k+), building views from scratch.

My experience was that I just needed to let couchdb build the view. It  
can take several minutes, and the CPU usage will be high. You should  
see both the beam and couchjs processes working while the view is  
building. If you're accessing a view via Futon, it's likely the  
browser will time-out the request before the build is finished. The  
build process *will* continue on the server side, though. If you let  
the build finish, the next time you query the view, it will return the  
data immediately.

To mitigate this problem, I'm now updating the view every time I do an  
insert (I bulk-add 20 records per minute). This only requires that the  
new data be added to the view, so building at this point is a short  
process.

(big thanks to the folks on #couchdb for helping me with this problem!)

--
Ed Finkler
http://funkatron.com
AIM: funka7ron
ICQ: 3922133
Skype: funka7ron


On Jul 31, 2008, at 5:10 PM, Demetrius Nunes wrote:

> Hi there,
>
> I was having a great time playing aroung with CouchDB. It seems like a
> perfect fit for a future system that we'll be building fairly soon.
>
> But then, I've just created a CouchDB database, importing 20.000  
> records
> from an old relational database into it.
>
> When I go into Futon, I can see the database is there, with 20.899  
> documents
> and 125.2 MB in size.
>
> Clicking on it, I can navigate thru the "All Documents" pretty  
> quickly (10
> documents per page).
>
> The problem is when I try to create a custom view. Just as I enter the
> custom view page in Futon, the server hangs and locks up my CPU at 90%
> usage. I waited several minutes for it to cool off but the process  
> was still
> there and I had no response at all.
>
> I then tried to create a view programatically, using REST/JSON and I  
> get the
> same result.
>
> I am running CouchDB 0.8.0 on Ubuntu 8.0.4.
>
> Is CouchDB not ready for a dataset of this size yet?
>
> Thanks and best regards,
> Dema
>
> -- 
> ____________________________
> http://www.demetriusnunes.com


Mime
View raw message