Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Message-Id: <E475A93E-2B29-4E51-837D-B1E8C02A6678@apache.org>
From: Damien Katz <damien@apache.org>
To: user@couchdb.apache.org
In-Reply-To: <20090429221138.GA22995@translab.its.uci.edu>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: Re: how much stuff can one stuff into a CouchDB?
Date: Wed, 29 Apr 2009 15:22:36 -0700
References: <20090429221138.GA22995@translab.its.uci.edu>

I think the total size of the data isn't a problem, but each access  
will require loading up the whole document into memory and 25 megs of  
data to read or update a small bit data, which can be quite  
inefficient. Views will be slow to update for each document change  
too, as it needs to load the whole thing into memory and serialize to  
the view engine, etc.

If the documents can be broken up into smaller updatable units, then  
things will work more smoothly generally, but still somewhat slow when  
building views. If the data can be stored as binary attachment, with  
just some meta data about the files stored in the JSON, views will be  
much more efficient.

-Damien

On Apr 29, 2009, at 3:11 PM, James Marca wrote:

> Hi All,
>
> On the Wiki, the FAQ says:
> Q: How Much Stuff can I Store in CouchDB?
> A: With node partitioning, virtually unlimited. For a single database
>   instance, the practical scaling limits aren't yet known.
>
> Is there some more recent guidance on this?  I read the wiki pages
> "Configuring distributed systems", "Partitioning proposal", and "HTTP
> Bulk Document API", but as far as I tell, node partitioning isn't
> implemented yet (right? things are moving really fast around here!).
> I have about 70G of gzipped files (about 3,000 files) that I need to
> unzip. convert to json, and store.  Unzipping explodes each file by
> about a factor of 7.  I expect that adding the JSON structure will
> increase the data size even more.  I read in an earlier posting that
> compacting the database will compress it back down significantly, but
> still, that's a big database file.
>
> I also have the option to break up the data into 9 logical chunks, but
> if I don't have to do it, I'd rather not.
>
> Anybody have any advice or experience with really big databases?
>
> Regards,
> James
> -- 
> James E. Marca
> Researcher
> Institute of Transportation Studies
> AIRB Suite 4000
> University of California
> Irvine, CA 92697-3600
> jmarca@translab.its.uci.edu
>
>
> -- 
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>