incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Demetrius Nunes" <demetriusnu...@gmail.com>
Subject Re: Is it possible to evaluate a view on a 20.000 documents database?
Date Thu, 31 Jul 2008 22:38:03 GMT
The view I am trying to create is really simple:

function(doc) {
  if
(doc.classe_id.match(/8a8090a20075ffba010075ffbed600028a8090a20075ffba010075ffbf7200c48a8090a20075ffba010075ffbf7200d9/))
    emit(doc.id, doc);
}

It's being applied to a 20.000 documents dataset and I've already waited
several minutes until the CPU cooled off, but to my surprise, the view is
still taking a long time to respond when I try to run it. Ive never actually
got a result out of it...

Am I doing something wrong?

Also, what are the performance goals for view-related operations like these
on bigger datasets (I consider a 20.000 document dataset fairly small) for
CouchDB? What shoud we expect for 1.0 ?

If it's not possible to evaluate views on these kinds of datasets in a few
seconds, then it would be huge deal-breaker for me. And I'd have to consider
using something like Sesame RDF database, but I really like CouchDB much
better.

Cheers,
Dema

On Thu, Jul 31, 2008 at 7:14 PM, Chris Anderson <jchris@grabb.it> wrote:

> If your view is complex, and you have many (100k+) records (and the
> emitted row size is large) views could take hours to generate on a
> Core Duo MacBook. Let them generate overnight, and in the morning the
> queries will be very fast.
>
> On Thu, Jul 31, 2008 at 2:44 PM, Ed Finkler <funkatron@gmail.com> wrote:
> > I have been working with a very similar problem, actually. A large set of
> > records (40k+), building views from scratch.
> >
> > My experience was that I just needed to let couchdb build the view. It
> can
> > take several minutes, and the CPU usage will be high. You should see both
> > the beam and couchjs processes working while the view is building. If
> you're
> > accessing a view via Futon, it's likely the browser will time-out the
> > request before the build is finished. The build process *will* continue
> on
> > the server side, though. If you let the build finish, the next time you
> > query the view, it will return the data immediately.
> >
> > To mitigate this problem, I'm now updating the view every time I do an
> > insert (I bulk-add 20 records per minute). This only requires that the
> new
> > data be added to the view, so building at this point is a short process.
> >
> > (big thanks to the folks on #couchdb for helping me with this problem!)
> >
> > --
> > Ed Finkler
> > http://funkatron.com
> > AIM: funka7ron
> > ICQ: 3922133
> > Skype: funka7ron
> >
> >
> > On Jul 31, 2008, at 5:10 PM, Demetrius Nunes wrote:
> >
> >> Hi there,
> >>
> >> I was having a great time playing aroung with CouchDB. It seems like a
> >> perfect fit for a future system that we'll be building fairly soon.
> >>
> >> But then, I've just created a CouchDB database, importing 20.000 records
> >> from an old relational database into it.
> >>
> >> When I go into Futon, I can see the database is there, with 20.899
> >> documents
> >> and 125.2 MB in size.
> >>
> >> Clicking on it, I can navigate thru the "All Documents" pretty quickly
> (10
> >> documents per page).
> >>
> >> The problem is when I try to create a custom view. Just as I enter the
> >> custom view page in Futon, the server hangs and locks up my CPU at 90%
> >> usage. I waited several minutes for it to cool off but the process was
> >> still
> >> there and I had no response at all.
> >>
> >> I then tried to create a view programatically, using REST/JSON and I get
> >> the
> >> same result.
> >>
> >> I am running CouchDB 0.8.0 on Ubuntu 8.0.4.
> >>
> >> Is CouchDB not ready for a dataset of this size yet?
> >>
> >> Thanks and best regards,
> >> Dema
> >>
> >> --
> >> ____________________________
> >> http://www.demetriusnunes.com
> >
> >
>
>
>
> --
> Chris Anderson
> http://jchris.mfdz.com
>



-- 
____________________________
http://www.demetriusnunes.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message