Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (athena.apache.org: domain of mike@cloudant.com designates
 209.85.212.172 as permitted sender)
Message-Id: <A207EFF6-EE1E-4429-8618-506D08E497F4@cloudant.com>
From: Mike Miller <mike@cloudant.com>
To: "user@couchdb.apache.org" <user@couchdb.apache.org>
In-Reply-To: <4D107979.5030805@bclary.com>
Content-Type: text/plain;
	charset=us-ascii;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (iPhone Mail 7E18)
Subject: Re: CouchDB becoming unusable as Database/Views increase in size.
Date: Tue, 21 Dec 2010 05:12:43 -0800
References: <4D107979.5030805@bclary.com>

Hi, bigcouch was built to address those exact needs. -Mike

On Dec 21, 2010, at 1:55 AM, Bob Clary <bob@bclary.com> wrote:

> Hi all,
>
> I've been using CouchDB to track the results of testing Firefox and  
> have found that as the database and view sizes have increased  
> CouchDB is becoming less and less viable as a solution going  
> forward. I don't wish to switch to a different database at this time  
> but may not have a choice.
>
> Let me say that I have looked at Jira and found others with similar  
> issues although issues have mostly been resolved as invalid or  
> already fixed. I do admit that I have a hard time navigating Jira,  
> so it is entirely possible I've missed already filed issues. I am  
> not sending this email in a threatening fashion that I've seen many  
> times in bugzilla where a user says "Fix this or I'm leaving!", but  
> in a plea for some help in finding, filing or fixing the appropriate  
> Jira issues which need attention.
>
> My database currently has a compacted size of about 37G and contains  
> a bit over 9 million documents. You can see examples of the view  
> documents in the error log I attached to <https://issues.apache.org/jira/browse/COUCHDB-970 
> >.
>
> I am currently using CouchDB 1.0.1 on Centos5 64bit vm with 2CPU and  
> 4G RAM running Erlang R14B and configured to use the 64bit js-devel  
> libraries. I temporarily tried to use CouchDB 1.0.x to pick up the  
> fix for <https://issues.apache.org/jira/browse/COUCHDB-926> which  
> was causing me problems but had to revert to 1.0.1 due to crashes  
> upon view compaction completion.
>
> Currently, my main issues are:
>
> Slow View generation: Recreating views from scratch is very slow. It  
> can take me over 24 hours to recreate some of the larger views.  
> Combined with the need to immediately compact them (see Large  
> Initial View sizes) recreating views can take my application offline  
> for users for more than a day. Trying to switch to 1.0.x and back  
> and having to regenerate views after out of space conditions has led  
> to my application being unavailable for most of a week.
>
> Large Initial View sizes: Several of my views are initially created  
> with sizes which are 10-20 times the size of the compacted view. For  
> example, I have one view which when initially created can take 95G  
> but when compacted uses less than 5G. This has caused several out of  
> disk space conditions when I've had to regenerate views for the  
> database. I know commodity disks are relatively cheap these days,  
> but due to my current hosting environment I am using relatively  
> expensive networked storage. Asking for sufficient storage for my  
> expected database size was difficult enough, but asking for 10 or  
> more times that amount just to deal with temporary explosive view  
> sizes is probably a non-starter.
>
> CouchDB 1.0.x: My experience with attempting to use the 1.0.x branch  
> was a failure due to the crashing immediately upon view compaction  
> completion which caused the views to begin indexing from scratch.
>
> I would appreciate it if you would let me know if some of these are  
> known issues which have already been filed in Jira or if it would be  
> helpful to file new issues and what additional information I can  
> provide to help get these issues resolved.
>
> I can also help in making newer releases of SpiderMonkey 1.7  
> available and to help get SpiderMonkey 1.8 and later released if  
> that will help the JavaScript performance issues CouchDB may be  
> facing.
>
> bc
>