Mailing-List: contact couchdb-user-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: couchdb-user@incubator.apache.org
Received-SPF: pass (athena.apache.org: domain of jchris@gmail.com designates
 74.125.46.155 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:sender:to:subject:in-reply-to:mime-version
         :content-type:content-transfer-encoding:content-disposition
         :references:x-google-sender-auth;
        b=saK60/Jiyc9V5NcdTI/ULEJNDb76kGjO4EXEVGbDLuTOt2JmzzlNVW31haJH/N+bNB
         jiuYzQ2RyQY99kKUpNplGlb/7s0Bcrm1bz/V5WIV2SxroVgzE8CedgKRz0S6r0GKaI7h
         E5bllLNSPO7C2U8Ynsq0XiVD3ZAwmGIi85y74=
Message-ID: <e282921e0807311514s1e09ff40j3d526b7c12cf92d3@mail.gmail.com>
Date: Thu, 31 Jul 2008 15:14:30 -0700
From: "Chris Anderson" <jchris@grabb.it>
Sender: jchris@gmail.com
To: couchdb-user@incubator.apache.org
Subject: Re: Is it possible to evaluate a view on a 20.000 documents database?
In-Reply-To: <06C596D4-E5D1-4161-BFF1-D3A40FFB4627@gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <4aa4f4d60807311410m6077d984x83cb3b5d4e8c6f17@mail.gmail.com>
	 <06C596D4-E5D1-4161-BFF1-D3A40FFB4627@gmail.com>

If your view is complex, and you have many (100k+) records (and the
emitted row size is large) views could take hours to generate on a
Core Duo MacBook. Let them generate overnight, and in the morning the
queries will be very fast.

On Thu, Jul 31, 2008 at 2:44 PM, Ed Finkler <funkatron@gmail.com> wrote:
> I have been working with a very similar problem, actually. A large set of
> records (40k+), building views from scratch.
>
> My experience was that I just needed to let couchdb build the view. It can
> take several minutes, and the CPU usage will be high. You should see both
> the beam and couchjs processes working while the view is building. If you're
> accessing a view via Futon, it's likely the browser will time-out the
> request before the build is finished. The build process *will* continue on
> the server side, though. If you let the build finish, the next time you
> query the view, it will return the data immediately.
>
> To mitigate this problem, I'm now updating the view every time I do an
> insert (I bulk-add 20 records per minute). This only requires that the new
> data be added to the view, so building at this point is a short process.
>
> (big thanks to the folks on #couchdb for helping me with this problem!)
>
> --
> Ed Finkler
> http://funkatron.com
> AIM: funka7ron
> ICQ: 3922133
> Skype: funka7ron
>
>
> On Jul 31, 2008, at 5:10 PM, Demetrius Nunes wrote:
>
>> Hi there,
>>
>> I was having a great time playing aroung with CouchDB. It seems like a
>> perfect fit for a future system that we'll be building fairly soon.
>>
>> But then, I've just created a CouchDB database, importing 20.000 records
>> from an old relational database into it.
>>
>> When I go into Futon, I can see the database is there, with 20.899
>> documents
>> and 125.2 MB in size.
>>
>> Clicking on it, I can navigate thru the "All Documents" pretty quickly (10
>> documents per page).
>>
>> The problem is when I try to create a custom view. Just as I enter the
>> custom view page in Futon, the server hangs and locks up my CPU at 90%
>> usage. I waited several minutes for it to cool off but the process was
>> still
>> there and I had no response at all.
>>
>> I then tried to create a view programatically, using REST/JSON and I get
>> the
>> same result.
>>
>> I am running CouchDB 0.8.0 on Ubuntu 8.0.4.
>>
>> Is CouchDB not ready for a dataset of this size yet?
>>
>> Thanks and best regards,
>> Dema
>>
>> --
>> ____________________________
>> http://www.demetriusnunes.com
>
>


-- 
Chris Anderson
http://jchris.mfdz.com