From user-return-20055-apmail-couchdb-user-archive=couchdb.apache.org@couchdb.apache.org Thu Mar 15 13:32:51 2012 Return-Path: X-Original-To: apmail-couchdb-user-archive@www.apache.org Delivered-To: apmail-couchdb-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B5A2E9A5D for ; Thu, 15 Mar 2012 13:32:51 +0000 (UTC) Received: (qmail 32249 invoked by uid 500); 15 Mar 2012 13:32:50 -0000 Delivered-To: apmail-couchdb-user-archive@couchdb.apache.org Received: (qmail 32224 invoked by uid 500); 15 Mar 2012 13:32:50 -0000 Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@couchdb.apache.org Delivered-To: mailing list user@couchdb.apache.org Received: (qmail 32216 invoked by uid 99); 15 Mar 2012 13:32:49 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Mar 2012 13:32:49 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [209.85.220.180] (HELO mail-vx0-f180.google.com) (209.85.220.180) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Mar 2012 13:32:44 +0000 Received: by vcbfl10 with SMTP id fl10so4435779vcb.11 for ; Thu, 15 Mar 2012 06:32:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:content-transfer-encoding:x-gm-message-state; bh=k5UtdRZKUvkP2NlCHoukoDtmdDX5hLjwuhRec34cg5I=; b=X4hf7NOb0WWsVfFp1VCY248BWvVdm2lBtqpnFMlOCmIANh2F238e1C/MlmHwybHWA0 J5PSJDOgTJF1idcbj5h9fpaOFMS+p1F/dUSvbL65Yzt+Fla0gXgQUxUecrjNaMRseoEm 0QaNLIlUge0MmNYfB+3/PSiY2iTjh8tgv2E+BVbG4xvKdurWrvOSFOGjFgU3oQnI2Xq5 vTynsUkAwQvX6eO2ZVHOS0ZOWxSuobfIKzQUjYF5Ky1qP2xZvopmMTV/raZd0cBMKy5/ GpqPNr0W7bZnjT89xp9M4l3JTo2hNKY2kdZbbcLua8HpTxfaaJ7RTU3PxYqsgL9bsk1y ZPIQ== Received: by 10.52.70.84 with SMTP id k20mr1918935vdu.67.1331818343161; Thu, 15 Mar 2012 06:32:23 -0700 (PDT) MIME-Version: 1.0 Received: by 10.220.156.212 with HTTP; Thu, 15 Mar 2012 06:32:03 -0700 (PDT) In-Reply-To: References: From: Jason Smith Date: Thu, 15 Mar 2012 13:32:03 +0000 Message-ID: Subject: Re: Creating a database with lots of documents and updating a view To: user@couchdb.apache.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Gm-Message-State: ALoCoQmXN7hfITlHtCU2Qwz8gNBFlAfxgY11hPspt9JZFUiYL7+Gx4mAwxOXwp66bBFBuW4OZRNa X-Virus-Checked: Checked by ClamAV on apache.org On Thu, Mar 15, 2012 at 1:21 PM, Daniel Gonzalez wr= ote: > Hi CGS, > > As you can see from the other threads that I have opened, the problem see= ms > to be threefold: > > =C2=A0 1. The bigger the database gets, the more size couchdb needs *per > =C2=A0 document* > =C2=A0 2. The bigger the database gets, the more time it takes to insert = new > =C2=A0 documents > =C2=A0 3. The bigger the database gets, the longer it gets to generate th= e > =C2=A0 views. The dependency is not lineal. Number 1 and 2 are true and *should* show logarithmically-increasing costs per document (i.e. not too bad). 500k updates per day is 6 updates per second which CouchDB can easily maintain and index. In that light, would you agree that you are currently enduring the long sunk cost of adding and indexing all those documents? (By comparison, mkfs.ext3 is painfully slow for large volumes, but we don't mind because we know it's a one-off cost.) Having said all of that, this is one of the problems BigCouch solves. It is pretty much API-compatible with CouchDB so you might investigate that option too. In my opinion, roughly speaking, you get better performance at a cost of a little more operational (sysadmin) work. --=20 Iris Couch