Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of tmnichols@gmail.com designates
 209.85.219.166 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=QueBpSPL7WzmXpVR56IiWn+VLBEKt8PmACYLFzPai/MLpJr0ezM4S4DsXoTX/fzLmQ
         6Q+o5LHsTBDUnZkvHoniZI9PhaXaDZATizGBrC1hnqjpYMuibIo5WP8+ky6kZoPh44ZD
         J0ID+Crk7YXCgs2N84WxNh4f5Q3XYY+Z3H3/4=
MIME-Version: 1.0
In-Reply-To: <49FF5E3E.8020604@proven-corporation.com>
References: <d8cb87ee0905040840i3929889fn86ce94018471ba3@mail.gmail.com>
	 <49FF5E3E.8020604@proven-corporation.com>
Date: Tue, 5 May 2009 12:59:14 -0400
Message-ID: <d8cb87ee0905050959w78167b47p1efee8781bbfda13@mail.gmail.com>
Subject: Re: Insert performance
From: Tom Nichols <tmnichols@gmail.com>
To: user@couchdb.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Thanks Jason.

I removed the EBS volume and things seemed a little faster.  CouchDB
is actually saturating the CPU now, so it looks like I'm getting
"accurate" benchmarks now.  i.e. I'm actually exercising CouchDB and
I'm not limited by some hardware/OS latency.

So I had 10 concurrent ruby processes performing batch inserts into
two test databases.  After 2 hours, I have 1.1M documents (1.5 GB) in
each database, for a total of 2GB and 2.2 million documents.  Does
this still seem slow?

Thanks.
-Tom


On Mon, May 4, 2009 at 5:29 PM, Jason Smith <jhs@proven-corporation.com> wr=
ote:
> Tom Nichols wrote:
>>
>> Hi, I have some questions about insert performance.
>>
>> I have a single CouchDB 0.9.0 node running on small EC2 instance. =A0I
>> attached a huge EBS volume to it and mounted it where CouchDB's data
>> files are stored. =A0I fired up about ruby scripts running inserts and
>> after a weekend I only have about 30GB/ 12M rows of data... =A0Which
>> seems small. =A0'top' tells me that my CPU is only about 30% utilized.
>>
>> Any idea what I might be doing wrong? =A0I pretty much just followed
>> these instructions:
>> http://wiki.apache.org/couchdb/Getting_started_with_Amazon_EC2
>
> Hi, Tom. =A0I believe I read somewhere before that the smallest EC2 insta=
nces
> have a slower and/or higher-latency connection to EBS, so you might want =
to
> consider a large instance, or maybe even a high-memory small instance and
> see whether you get better "hardware" performance.
>
> Although strangely, when googling it, the first article I found says that
> their benchmarks found no difference between EBS or even the ephemeral
> filesystem.
>
> http://www.paessler.com/blog/2009/04/07/prtg-7/monitoring-cloud-performan=
ce-with-prtg-comparing-disk-speed-for-instance-stores-and-ebs-volumes-on-am=
azon-ec2/
>
> On the other hand, here is a forum posting and a random benchmark indicat=
ing
> that more expensive instances get better throughput:
>
> http://developer.amazonwebservices.com/connect/message.jspa?messageID=3D1=
25197
> http://blog.getasysadmin.com/2009/02/mysql-benchmarks-using-amazon-ec2.ht=
ml
>
> --
> Jason Smith
> Proven Corporation
> Bangkok, Thailand
> http://www.proven-corporation.com
>