Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (athena.apache.org: local policy)
Date: Wed, 29 Apr 2009 15:11:38 -0700
From: James Marca <jmarca@translab.its.uci.edu>
To: user@couchdb.apache.org
Subject: how much stuff can one stuff into a CouchDB?
Message-ID: <20090429221138.GA22995@translab.its.uci.edu>
Mail-Followup-To: user@couchdb.apache.org
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.4.1i

Hi All, 

On the Wiki, the FAQ says:
Q: How Much Stuff can I Store in CouchDB?
A: With node partitioning, virtually unlimited. For a single database
   instance, the practical scaling limits aren't yet known. 

Is there some more recent guidance on this?  I read the wiki pages
"Configuring distributed systems", "Partitioning proposal", and "HTTP
Bulk Document API", but as far as I tell, node partitioning isn't
implemented yet (right? things are moving really fast around here!).
I have about 70G of gzipped files (about 3,000 files) that I need to
unzip. convert to json, and store.  Unzipping explodes each file by
about a factor of 7.  I expect that adding the JSON structure will
increase the data size even more.  I read in an earlier posting that
compacting the database will compress it back down significantly, but
still, that's a big database file.

I also have the option to break up the data into 9 logical chunks, but
if I don't have to do it, I'd rather not.

Anybody have any advice or experience with really big databases?

Regards, 
James
-- 
James E. Marca
Researcher
Institute of Transportation Studies
AIRB Suite 4000
University of California
Irvine, CA 92697-3600
jmarca@translab.its.uci.edu


-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.