Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of jbellis@gmail.com designates
 209.85.220.172 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=wQXdceF+w2JzJOlpJCulr9aXFzKoTn+QtOYS3AekC/cs90ZATENV+F5dwCHgtlc17w
         eT4lMpfjoHCgqPvq1LO26BWnQHban01FmHOnKTuRdFfPMnhg9z9kJA6b9Usn5+XjlIQO
         H1Cwe0otVbwTADgnX8CelH5vPl330x4Im5MrY=
MIME-Version: 1.0
In-Reply-To: <4DAF3A7C.3020903@alex.otherinbox.com>
References: <4DAF3A7C.3020903@alex.otherinbox.com>
From: Jonathan Ellis <jbellis@gmail.com>
Date: Wed, 20 Apr 2011 17:12:31 -0500
Message-ID: <BANLkTikdH3y4VE3b0H3G4jG0Gq7nM6G6dw@mail.gmail.com>
Subject: Re: Ec2 Stress Results
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

A few months ago I was seeing 12k writes/s on a single EC2 XL. So
something is wrong.

My first suspicion is that your client node may be the bottleneck.

On Wed, Apr 20, 2011 at 2:56 PM, Alex Araujo
<cassandra-users@alex.otherinbox.com> wrote:
> Does anyone have any Ec2 benchmarks/experiences they can share?=A0 I am t=
rying
> to get a sense for what to expect from a production cluster on Ec2 so tha=
t I
> can compare my application's performance against a sane baseline.=A0 What=
 I
> have done so far is:
>
> 1. Lunched a 4 node cluster of m1.xlarge instances in the same availabili=
ty
> zone using PyStratus (https://github.com/digitalreasoning/PyStratus).=A0 =
Each
> node has the following specs (according to Amazon):
> 15 GB memory
> 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each)
> 1,690 GB instance storage
> 64-bit platform
>
> 2. Changed the default PyStratus directories in order to have commit logs=
 on
> the root partition and data files on ephemeral storage:
> commitlog_directory: /var/cassandra-logs
> data_file_directories: [/mnt/cassandra-data]
>
> 2. Gave each node 10GB of MAX_HEAP; 1GB HEAP_NEWSIZE in
> conf/cassandra-env.sh
>
> 3. Ran `contrib/stress/bin/stress -d node1,..,node4 -n 10000000 -t 100` o=
n a
> separate m1.large instance:
> total,interval_op_rate,interval_key_rate,avg_latency,elapsed_time
> ...
> 9832712,7120,7120,0.004948514851485148,842
> 9907616,7490,7490,0.0043189949802413755,852
> 9978357,7074,7074,0.004560353967289125,863
> 10000000,2164,2164,0.004065933558194335,867
>
> 4. Truncated Keyspace1.Standard1:
> # /usr/local/apache-cassandra/bin/cassandra-cli -host localhost -port 916=
0
> Connected to: "Test Cluster" on x.x.x.x/9160
> Welcome to cassandra CLI.
>
> Type 'help;' or '?' for help. Type 'quit;' or 'exit;' to quit.
> [default@unknown] use Keyspace1;
> Authenticated to keyspace: Keyspace1
> [default@Keyspace1] truncate Standard1;
> null
>
> 5. Expanded the cluster to 8 nodes using PyStratus and sanity checked usi=
ng
> nodetool:
> # /usr/local/apache-cassandra/bin/nodetool -h localhost ring
> Address=A0=A0=A0=A0=A0=A0=A0=A0 Status State=A0=A0 Load=A0=A0=A0=A0=A0=A0=
=A0=A0=A0=A0=A0 Owns
> Token
> x.x.x.x=A0 Up=A0=A0=A0=A0 Normal=A0 1.3 GB=A0=A0=A0=A0=A0=A0=A0=A0=A0 12.=
50%
> 21267647932558653966460912964485513216
> x.x.x.x=A0=A0 Up=A0=A0=A0=A0 Normal=A0 3.06 GB=A0=A0=A0=A0=A0=A0=A0=A0 12=
.50%
> 42535295865117307932921825928971026432
> x.x.x.x=A0=A0=A0=A0 Up=A0=A0=A0=A0 Normal=A0 1.16 GB=A0=A0=A0=A0=A0=A0=A0=
=A0 12.50%
> 63802943797675961899382738893456539648
> x.x.x.x=A0=A0 Up=A0=A0=A0=A0 Normal=A0 2.43 GB=A0=A0=A0=A0=A0=A0=A0=A0 12=
.50%
> 85070591730234615865843651857942052864
> x.x.x.x=A0=A0 Up=A0=A0=A0=A0 Normal=A0 1.22 GB=A0=A0=A0=A0=A0=A0=A0=A0 12=
.50%
> 106338239662793269832304564822427566080
> x.x.x.x=A0=A0=A0 Up=A0=A0=A0=A0 Normal=A0 2.74 GB=A0=A0=A0=A0=A0=A0=A0=A0=
 12.50%
> 127605887595351923798765477786913079296
> x.x.x.x=A0=A0=A0 Up=A0=A0=A0=A0 Normal=A0 1.22 GB=A0=A0=A0=A0=A0=A0=A0=A0=
 12.50%
> 148873535527910577765226390751398592512
> x.x.x.x=A0=A0 Up=A0=A0=A0=A0 Normal=A0 2.57 GB=A0=A0=A0=A0=A0=A0=A0=A0 12=
.50%
> 170141183460469231731687303715884105728
>
> 6. Ran `contrib/stress/bin/stress -d node1,..,node8 -n 10000000 -t 100` o=
n a
> separate m1.large instance again:
> total,interval_op_rate,interval_key_rate,avg_latency,elapsed_time
> ...
> 9880360,9649,9649,0.003210443956226165,720
> 9942718,6235,6235,0.003206934154398794,731
> 9997035,5431,5431,0.0032615939761032457,741
> 10000000,296,296,0.002660033726812816,742
>
> In a nutshell, 4 nodes inserted at 11,534 writes/sec and 8 nodes inserted=
 at
> 13,477 writes/sec.
>
> Those numbers seem a little low to me, but I don't have anything to compa=
re
> to.=A0 I'd like to hear others' opinions before I spin my wheels with wit=
h
> number of nodes, threads,=A0 memtable, memory, and/or GC settings.=A0 Che=
ers,
> Alex.
>


--=20
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com