Return-Path: Delivered-To: apmail-incubator-couchdb-user-archive@locus.apache.org Received: (qmail 7414 invoked from network); 3 Nov 2008 11:01:09 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 3 Nov 2008 11:01:09 -0000 Received: (qmail 8477 invoked by uid 500); 3 Nov 2008 11:01:14 -0000 Delivered-To: apmail-incubator-couchdb-user-archive@incubator.apache.org Received: (qmail 8450 invoked by uid 500); 3 Nov 2008 11:01:14 -0000 Mailing-List: contact couchdb-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: couchdb-user@incubator.apache.org Delivered-To: mailing list couchdb-user@incubator.apache.org Received: (qmail 8439 invoked by uid 99); 3 Nov 2008 11:01:14 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Nov 2008 03:01:14 -0800 X-ASF-Spam-Status: No, hits=0.2 required=10.0 tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [209.68.5.9] (HELO relay00.pair.com) (209.68.5.9) by apache.org (qpsmtpd/0.29) with SMTP; Mon, 03 Nov 2008 10:59:58 +0000 Received: (qmail 13503 invoked from network); 3 Nov 2008 10:59:38 -0000 Received: from 96.33.90.152 (HELO ?192.168.1.200?) (96.33.90.152) by relay00.pair.com with SMTP; 3 Nov 2008 10:59:38 -0000 X-pair-Authenticated: 96.33.90.152 Message-Id: <09A61EA1-ECC6-49A5-AFE9-B5F7C48A63B7@apache.org> From: Damien Katz To: couchdb-user@incubator.apache.org In-Reply-To: <57E1752F-5513-4F8A-97E0-8D18067D511E@apache.org> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v929.2) Subject: Re: Largest CouchDB dbs? Date: Mon, 3 Nov 2008 05:59:38 -0500 References: <8C1EE3D1-D62F-4C6A-A859-196297E09C3D@apache.org> <0E689907656A4B499E477CBB9A24044707FAF6D0@sryulwis0comx01.coradiant.com> <57E1752F-5513-4F8A-97E0-8D18067D511E@apache.org> X-Mailer: Apple Mail (2.929.2) X-Virus-Checked: Checked by ClamAV on apache.org On Nov 3, 2008, at 4:38 AM, Jan Lehnardt wrote: > > On Nov 3, 2008, at 05:53, Jonathan Ginter wrote: > >> I have a similar issue. I am interested in using CouchDB to host a >> 200+ GB database that will receive well over 200 million documents >> per day. Moreover, the data must roll out - i.e., constant >> background purging - and also support UI queries. And this is just >> a starting point to match the abilities of the relational database >> we are already running. I will want the DB to scale up from there. >> >> If there is no hope of the CouchDB being able to handle all of that >> - regardless of how many machines we deploy - I would like to know >> that now before I look any further into this project. > >> Does anyone have a reasonable idea about whether CouchDB will be >> capable of such massive scalability or how many machines it would >> take to scale that large? > > This sounds like a scenario that CouchDB will ultimately be able to > handle nicely. I don't think we can give out any guarantees about when > an how this will be the case. Maintaining a 200+GB data set would > require > quite some hand-wiring at the moment. > > >> I would appreciate any feedback that anyone might have on this. > > I think Damien can chime in here :) Damien? > This is definitely well within what couchdb should be able to do once partitioning is in place. I'm not really working on this yet, but there are a lot of people and companies interested in seeing the partitioning work done. So maybe some progress will be made soon. -Damien