Mailing-List: contact couchdb-user-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: couchdb-user@incubator.apache.org
Received-SPF: pass (athena.apache.org: domain of schickb@gmail.com designates
 64.233.166.179 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject:references
         :in-reply-to:content-type:content-transfer-encoding;
        b=Hyt+WlKSXoXeuGvZWBpG7CR+G7GpqoOc2Qz0wLORiZhTZTL+h0BRIQZOMp/34Xk3R0
         oJdrjgz0bNtgtSomsHrt9ZJwEqiuR6+httca/p3bHFfFAtDpFgOMFeHDEtp6VQMU0lMB
         iHVHuBtIw61k3Ag4sX1OMUayzDn1jR4OYRecQ=
Message-ID: <485832F7.5080009@gmail.com>
Date: Tue, 17 Jun 2008 14:56:07 -0700
From: Brad Schick <schickb@gmail.com>
User-Agent: Thunderbird 2.0.0.14 (X11/20080505)
MIME-Version: 1.0
To: couchdb-user@incubator.apache.org
Subject: Re: Bad write performance
References: <48580C64.40806@gmail.com>
 <A1C5C173-0BC9-4CB0-8873-7070FA0117D0@variogr.am>
In-Reply-To: <A1C5C173-0BC9-4CB0-8873-7070FA0117D0@variogr.am>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

On 06/17/2008 12:57 PM, Brian Whitman wrote:
>
> On Jun 17, 2008, at 3:11 PM, Brad Schick wrote:
>
>> I am seeing very poor write performance running CouchDB on Ubuntu Linux
>> 8.04. I built CouchDB from source about a week ago, and I am using the
>> Python wrapper to access it all on localhost.
>>
>> I am trying to add over 90,000 documents (each is about 400 bytes) and
>> finding that I can only add about 16 documents per second. And while
>> this is happening, my CPU is about 75% idle.
>
> Are you using the update method in python (which uses _bulk_docs) ?
> You need the svn version of the python wrapper for it.
>
> On a slow machine (dual proc 1.8 GHz) using the python wrapper I added
> 100K docs 10K at a time, each with a random string, uuid, float and
> int in 180 seconds (roughly 550 documents/second.) One CPU was pegged
> at 100% with beam.smp during the update and the python client did not
> even rate on top during the update (it was just waiting for a response.)
>
> I couldn't increase my chunk size much past 10K -- couch would return
> a (very long) error if I tried 5 20K chunks for example.
>

I have the code from svn, but I haven't tried bulk uploading yet because
I've been primarily testing schema.py which currently only does single
document stores. Maybe I'll hook that up to bulk writes and see how it goes.

-Brad