From user-return-4988-apmail-couchdb-user-archive=couchdb.apache.org@couchdb.apache.org Tue May 26 22:11:10 2009 Return-Path: Delivered-To: apmail-couchdb-user-archive@www.apache.org Received: (qmail 61720 invoked from network); 26 May 2009 22:11:10 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 26 May 2009 22:11:10 -0000 Received: (qmail 79713 invoked by uid 500); 26 May 2009 22:11:22 -0000 Delivered-To: apmail-couchdb-user-archive@couchdb.apache.org Received: (qmail 79624 invoked by uid 500); 26 May 2009 22:11:22 -0000 Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@couchdb.apache.org Delivered-To: mailing list user@couchdb.apache.org Received: (qmail 79614 invoked by uid 99); 26 May 2009 22:11:22 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 May 2009 22:11:22 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of macfisherman@gmail.com designates 209.85.218.170 as permitted sender) Received: from [209.85.218.170] (HELO mail-bw0-f170.google.com) (209.85.218.170) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 May 2009 22:11:13 +0000 Received: by bwz18 with SMTP id 18so4962066bwz.11 for ; Tue, 26 May 2009 15:10:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=nFqFQKc21hgLZWn5WG8t5lsxvjHobz01tbOHwaUMV88=; b=P5Khbdz4pzuegLhLfrdG8vDV8ekSiQImXTF0SUZgo/sBU/1JSo+NVCtSD9sz8JsVQ4 +q/CdMuKWeipohSTDJH/JmLNbX31uWRwlZEwXV353xeH9hdHCFKXbSyv2tGS9zg4l9sW bUZSoU0CTwNGEconXk2hUdkXW9fLxqkRDEx78= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=gjJg2hqGlI+Bdh+hzxkB2HYJKzid0tnUQi/Tykl9Jlr6Zm3eoDtoXSEYUCRIUeAjQ1 S5Z7RnlnW0i54c8rMl4WgUnEHz9fJUh87bqsTgU2DcMD7bGNv4G9+2TVsqFVpQfPLp6S uLZJnUH+W+MHc/GriwOJ+t5IKWz3jyHw66ggI= MIME-Version: 1.0 Received: by 10.204.54.198 with SMTP id r6mr8516340bkg.191.1243375852413; Tue, 26 May 2009 15:10:52 -0700 (PDT) In-Reply-To: References: <45ae90370905261431s45669c59t680d256d9c600d5f@mail.gmail.com> Date: Tue, 26 May 2009 18:10:52 -0400 Message-ID: <45ae90370905261510i74fe4068ke9bdf2be0e9b4502@mail.gmail.com> Subject: Re: specifying an _id results in a much smaller DB? From: Jeff Macdonald To: user@couchdb.apache.org Content-Type: multipart/alternative; boundary=001636c5ba5901623d046ad800c4 X-Virus-Checked: Checked by ClamAV on apache.org --001636c5ba5901623d046ad800c4 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit On Tue, May 26, 2009 at 5:36 PM, Chris Anderson wrote: > On Tue, May 26, 2009 at 2:31 PM, Jeff Macdonald > wrote: > > Hi all, > > I've been experimenting with CouchDB. I'm use Net::CouchDB to batch > insert > > 20 docs at a time and I'm simply setting _id to a sequence that is > > incremented for each doc. For just over 9 million rows where each row is > > just 6 small fields the resulting DB is 3.4G. When I was letting CouchDB > set > > the _id, the resulting database was over 20G. The input source as a tab > > delimited file is just over 500MB. > > > > So is it normal for CouchDB to create such a large database file when it > > assigns document ids? > > > > yes, currently couchdb docids are random which means more of the btree > must be rewritten, than if they were concentrated, such as you see > with sequential ids. for high performance applications, sequential ids > is faster as well. > > Compacting may shrink your databases so they are roughly equal size. > You an trigger compaction from Futon. I'd be interested to see what > results you get. Well, it took over a day to do it before. I was however only inserting 10 docs at a time then. So, right now I'm not motivated to find out how well the compaction would be. :) -- Jeff Macdonald Ayer, MA --001636c5ba5901623d046ad800c4--