incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Damien Katz <dam...@apache.org>
Subject Re: Is it possible to evaluate a view on a 20.000 documents database?
Date Thu, 31 Jul 2008 22:45:57 GMT
I'm guessing you are running an old version of erlang (R11) known to  
have performance issues. Upgrade to the latest (R12 B-3 available from  
erlang.org),  The stuff in the packages (apt, macport etc) is usually  
outdated.

-Damien

On Jul 31, 2008, at 6:38 PM, Demetrius Nunes wrote:

> The view I am trying to create is really simple:
>
> function(doc) {
>  if
> (doc.classe_id.match(/ 
> 8a8090a20075ffba010075ffbed600028a8090a20075ffba010075ffbf7200c48a8090a20075ffba010075ffbf7200d9

> /))
>    emit(doc.id, doc);
> }
>
> It's being applied to a 20.000 documents dataset and I've already  
> waited
> several minutes until the CPU cooled off, but to my surprise, the  
> view is
> still taking a long time to respond when I try to run it. Ive never  
> actually
> got a result out of it...
>
> Am I doing something wrong?
>
> Also, what are the performance goals for view-related operations  
> like these
> on bigger datasets (I consider a 20.000 document dataset fairly  
> small) for
> CouchDB? What shoud we expect for 1.0 ?
>
> If it's not possible to evaluate views on these kinds of datasets in  
> a few
> seconds, then it would be huge deal-breaker for me. And I'd have to  
> consider
> using something like Sesame RDF database, but I really like CouchDB  
> much
> better.
>
> Cheers,
> Dema
>
> On Thu, Jul 31, 2008 at 7:14 PM, Chris Anderson <jchris@grabb.it>  
> wrote:
>
>> If your view is complex, and you have many (100k+) records (and the
>> emitted row size is large) views could take hours to generate on a
>> Core Duo MacBook. Let them generate overnight, and in the morning the
>> queries will be very fast.
>>
>> On Thu, Jul 31, 2008 at 2:44 PM, Ed Finkler <funkatron@gmail.com>  
>> wrote:
>>> I have been working with a very similar problem, actually. A large  
>>> set of
>>> records (40k+), building views from scratch.
>>>
>>> My experience was that I just needed to let couchdb build the  
>>> view. It
>> can
>>> take several minutes, and the CPU usage will be high. You should  
>>> see both
>>> the beam and couchjs processes working while the view is building.  
>>> If
>> you're
>>> accessing a view via Futon, it's likely the browser will time-out  
>>> the
>>> request before the build is finished. The build process *will*  
>>> continue
>> on
>>> the server side, though. If you let the build finish, the next  
>>> time you
>>> query the view, it will return the data immediately.
>>>
>>> To mitigate this problem, I'm now updating the view every time I  
>>> do an
>>> insert (I bulk-add 20 records per minute). This only requires that  
>>> the
>> new
>>> data be added to the view, so building at this point is a short  
>>> process.
>>>
>>> (big thanks to the folks on #couchdb for helping me with this  
>>> problem!)
>>>
>>> --
>>> Ed Finkler
>>> http://funkatron.com
>>> AIM: funka7ron
>>> ICQ: 3922133
>>> Skype: funka7ron
>>>
>>>
>>> On Jul 31, 2008, at 5:10 PM, Demetrius Nunes wrote:
>>>
>>>> Hi there,
>>>>
>>>> I was having a great time playing aroung with CouchDB. It seems  
>>>> like a
>>>> perfect fit for a future system that we'll be building fairly soon.
>>>>
>>>> But then, I've just created a CouchDB database, importing 20.000  
>>>> records
>>>> from an old relational database into it.
>>>>
>>>> When I go into Futon, I can see the database is there, with 20.899
>>>> documents
>>>> and 125.2 MB in size.
>>>>
>>>> Clicking on it, I can navigate thru the "All Documents" pretty  
>>>> quickly
>> (10
>>>> documents per page).
>>>>
>>>> The problem is when I try to create a custom view. Just as I  
>>>> enter the
>>>> custom view page in Futon, the server hangs and locks up my CPU  
>>>> at 90%
>>>> usage. I waited several minutes for it to cool off but the  
>>>> process was
>>>> still
>>>> there and I had no response at all.
>>>>
>>>> I then tried to create a view programatically, using REST/JSON  
>>>> and I get
>>>> the
>>>> same result.
>>>>
>>>> I am running CouchDB 0.8.0 on Ubuntu 8.0.4.
>>>>
>>>> Is CouchDB not ready for a dataset of this size yet?
>>>>
>>>> Thanks and best regards,
>>>> Dema
>>>>
>>>> --
>>>> ____________________________
>>>> http://www.demetriusnunes.com
>>>
>>>
>>
>>
>>
>> --
>> Chris Anderson
>> http://jchris.mfdz.com
>>
>
>
>
> -- 
> ____________________________
> http://www.demetriusnunes.com


Mime
View raw message