Return-Path: 
 <user-return-16705-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org>
X-Original-To: apmail-cassandra-user-archive@www.apache.org
Delivered-To: apmail-cassandra-user-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DCB054F67
	for <apmail-cassandra-user-archive@www.apache.org>;
 Thu, 12 May 2011 00:26:21 +0000 (UTC)
Received: (qmail 69181 invoked by uid 500); 12 May 2011 00:26:19 -0000
Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org
Received: (qmail 69150 invoked by uid 500); 12 May 2011 00:26:19 -0000
Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@cassandra.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@cassandra.apache.org>
List-Post: <mailto:user@cassandra.apache.org>
List-Id: <user.cassandra.apache.org>
Reply-To: user@cassandra.apache.org
Delivered-To: mailing list user@cassandra.apache.org
Received: (qmail 69142 invoked by uid 99); 12 May 2011 00:26:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 00:26:19 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.218.44] (HELO mail-yi0-f44.google.com) (209.85.218.44)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 00:26:12 +0000
Received: by yic13 with SMTP id 13so452666yic.31
        for <user@cassandra.apache.org>; Wed, 11 May 2011 17:25:51 -0700 (PDT)
Received: by 10.101.205.22 with SMTP id h22mr430398anq.14.1305159951615;
        Wed, 11 May 2011 17:25:51 -0700 (PDT)
Received: from alex.local (cpe-66-68-190-114.austin.res.rr.com
 [66.68.190.114])
        by mx.google.com with ESMTPS id c35sm412028anp.7.2011.05.11.17.25.50
        (version=SSLv3 cipher=OTHER);
        Wed, 11 May 2011 17:25:51 -0700 (PDT)
Message-ID: <4DCB290E.7070209@alex.otherinbox.com>
Date: Wed, 11 May 2011 19:25:50 -0500
From: Alex Araujo <cassandra-users@alex.otherinbox.com>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US;
 rv:1.9.2.17) Gecko/20110414 Thunderbird/3.1.10
MIME-Version: 1.0
To: user@cassandra.apache.org
Subject: Re: Ec2 Stress Results
References: <4DAF3A7C.3020903@alex.otherinbox.com>
 <BANLkTikdH3y4VE3b0H3G4jG0Gq7nM6G6dw@mail.gmail.com>
 <BANLkTi=wZApxnJ82J1bvbCc+pAOsny2QJw@mail.gmail.com>
 <4DC48E8D.2020501@alex.otherinbox.com>
 <BANLkTi=BK5CXY-3B3E9Ck+kGe6GNL=3Huw@mail.gmail.com>
 <4DC87186.3060806@alex.otherinbox.com>
 <BANLkTimSVRDD6-ZBzcMZ_RVjcCwRy4D9LQ@mail.gmail.com>
In-Reply-To: <BANLkTimSVRDD6-ZBzcMZ_RVjcCwRy4D9LQ@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 5/9/11 9:49 PM, Jonathan Ellis wrote:
> On Mon, May 9, 2011 at 5:58 PM, Alex Araujo<cassandra->>  How many
> replicas are you writing?
>> Replication factor is 3.
> So you're actually spot on the predicted numbers: you're pushing
> 20k*3=60k "raw" rows/s across your 4 machines.
>
> You might get another 10% or so from increasing memtable thresholds,
> but bottom line is you're right around what we'd expect to see.
> Furthermore, CPU is the primary bottleneck which is what you want to
> see on a pure write workload.
>
That makes a lot more sense.  I upgraded the cluster to 4 m2.4xlarge 
instances (68GB of RAM/8 CPU cores) in preparation for application 
stress tests and the results were impressive @ 200 threads per client:

+--------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
| Server Nodes | Client Nodes | --keep-going |   Columns    |    
Client    |    Total     |  Rep Factor  |  Test Rate   | Cluster Rate |
|              |              |              |              |   
Threads    |   Threads    |              |  (writes/s)  |  (writes/s)  |
+==============+==============+==============+==============+==============+==============+==============+==============+==============+
|      4       |      3       |      N       |   10000000   |     
200      |     600      |      3       |    44644     |    133931    |
+--------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+

The issue I'm seeing with app stress tests is that the rate will be 
comparable/acceptable at first (~100k w/s) and will degrade considerably 
(~48k w/s) until a flush and restart.  CPU usage will correspondingly be 
high at first (500-700%) and taper down to 50-200%.  My data model is 
pretty standard (<This> is pseudo-type information):

Users<Column>
"UserId<32CharHash>" : {
     "email<String>": "a@b.com",
     "first_name<String>": "John",
     "last_name<String>": "Doe"
}

UserGroups<SuperColumn>
"GroupId<UUID>": {
     "UserId<32CharHash>": {
         "date_joined<DateTime>": "2011-05-10 13:14.789",
         "date_left<DateTime>": "2011-05-11 13:14.789",
         "active<short>": "0|1"
     }
}

UserGroupTimeline<Column>
"GroupId<UUID>": {
     "date_joined<TimeUUID>": "UserId<32CharHash>"
}

UserGroupStatus<Column>
"CompositeId('GroupId<UUID>:UserId<32CharHash>')": {
     "active<short>": "0|1"
}

Every new User has a row in Users and a ColumnOrSuperColumn in the other 
3 CFs (total of 4 operations).  One notable difference is that the RAID0 
on this instance type (surprisingly) only contains two ephemeral volumes 
and appear a bit more saturated in iostat, although not enough to 
clearly stand out as the bottleneck.  Is the bottleneck in this scenario 
likely memtable flush and/or commitlog rotation settings?

RF = 2; ConsistencyLevel = One; -Xmx = 6GB; concurrent_writes: 64; all 
other settings are the defaults.  Thanks, Alex.