Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of markusklems@gmail.com
 designates 209.85.160.172 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:from:date
         :x-google-sender-auth:message-id:subject:to:content-type
         :content-transfer-encoding;
        b=aB+SHipRjR36TBHTYBsjpo6pH0S6nyYgybHRXporV785Js+xzyHTiJTefSd63rCLIE
         Z7374Lrd5UkTChKrJ+3AjPJ1B8KKAW/xNhZrhIYk1y87JBB8ObRr2fCVj5UB1q7Ccz71
         Hwb+dp+6QaYGQF5qU36T5pKrXbueB6s0kviMY=
MIME-Version: 1.0
Sender: markusklems@gmail.com
In-Reply-To: <FF350302-C298-4763-9A75-CBF22588C559@thelastpickle.com>
References: <AANLkTikxHf+4dk57XH6iJ8V8JGPRQMVifpBsJSEz4pN=@mail.gmail.com>
 <AANLkTimjseci72rxDgEzaSgbwodR18=8bPzku-4Wvs-o@mail.gmail.com>
 <FF350302-C298-4763-9A75-CBF22588C559@thelastpickle.com>
From: Markus Klems <markus@klems.eu>
Date: Sat, 19 Feb 2011 09:53:06 +0100
Message-ID: <AANLkTinY6zw-ktvQr1O4Cs4k2e=aJ0UN9q_StDUwykMW@mail.gmail.com>
Subject: Re: Benchmarking Cassandra with YCSB
To: user@cassandra.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi,

we sorted out the performance problems and tuned the cluster. In
particular, we identified the following weak spot in our setup:
ConcurrentReads and ConcurrentWrites was set to the default values
which were much too low for our setup. Now, we get some serious
numbers.

Thanks,

Markus

On Tue, Feb 15, 2011 at 9:09 PM, Aaron Morton <aaron@thelastpickle.com> wro=
te:
> Initial thoughts are you are overloading the cluster, are their any log l=
ines about dropping messages?
>
> What is the schema, what settings do you have in Cassandra yaml =C2=A0and=
 what are CF stats telling you? E.g. Are you switching Memtables too quickl=
y? What are the write latency numbers?
>
> Also 0.7 is much faster.
>
> Aaron
>
> On 16/02/2011, at 8:59 AM, Thibaut Britz <thibaut.britz@trendiction.com> =
wrote:
>
>> Cassandra is very CPU hungry so you might be hitting a CPU bottleneck.
>> What's your CPU usage during these tests?
>>
>>
>> On Tue, Feb 15, 2011 at 8:45 PM, Markus Klems <markus@klems.eu> wrote:
>>> Hi there,
>>>
>>> we are currently benchmarking a Cassandra 0.6.5 cluster with 3
>>> High-Mem Quadruple Extra Large EC2 nodes
>>> (http://aws.amazon.com/ec2/#instance) using Yahoo's YCSB tool
>>> (replication factor is 3, random partitioner). We assigned 32 GB RAM
>>> to the JVM and left 32 GB RAM for the Ubuntu Linux filesystem buffer.
>>> We also set the user count to a very large number via ulimit -u
>>> 999999.
>>>
>>> Our goal is to achieve max throughput by increasing YCSB's threadcount
>>> parameter (i.e. the number of parallel benchmarking client threads).
>>> However, this does only improve Cassandra throughput for low numbers
>>> of threads. If we move to higher threadcounts, throughput does not
>>> increase and even =C2=A0decreases. Do you have any idea why this is
>>> happening and possibly suggestions how to scale throughput to much
>>> higher numbers? Why is throughput hitting a wall, anyways? And where
>>> does the latency/throughput tradeoff come from?
>>>
>>> Here is our YCSB configuration:
>>> recordcount=3D300000
>>> operationcount=3D1000000
>>> workload=3Dcom.yahoo.ycsb.workloads.CoreWorkload
>>> readallfields=3Dtrue
>>> readproportion=3D0.5
>>> updateproportion=3D0.5
>>> scanproportion=3D0
>>> insertproportion=3D0
>>> threadcount=3D 500
>>> target =3D 10000
>>> hosts=3DEC2-1,EC2-2,EC2-3
>>> requestdistribution=3Duniform
>>>
>>> These are typical results for threadcount=3D1:
>>> Loading workload...
>>> Starting test.
>>> =C2=A00 sec: 0 operations;
>>> =C2=A010 sec: 11733 operations; 1168.28 current ops/sec; [UPDATE
>>> AverageLatency(ms)=3D0.64] [READ AverageLatency(ms)=3D1.03]
>>> =C2=A020 sec: 24246 operations; 1251.68 current ops/sec; [UPDATE
>>> AverageLatency(ms)=3D0.48] [READ AverageLatency(ms)=3D1.11]
>>>
>>> These are typical results for threadcount=3D10:
>>> 10 sec: 30428 operations; 3029.77 current ops/sec; [UPDATE
>>> AverageLatency(ms)=3D2.11] [READ AverageLatency(ms)=3D4.32]
>>> =C2=A020 sec: 60838 operations; 3041.91 current ops/sec; [UPDATE
>>> AverageLatency(ms)=3D2.15] [READ AverageLatency(ms)=3D4.37]
>>>
>>> These are typical results for threadcount=3D100:
>>> 10 sec: 29070 operations; 2895.42 current ops/sec; [UPDATE
>>> AverageLatency(ms)=3D20.53] [READ AverageLatency(ms)=3D44.91]
>>> =C2=A020 sec: 53621 operations; 2455.84 current ops/sec; [UPDATE
>>> AverageLatency(ms)=3D23.11] [READ AverageLatency(ms)=3D55.39]
>>>
>>> These are typical results for threadcount=3D500:
>>> 10 sec: 30655 operations; 3053.59 current ops/sec; [UPDATE
>>> AverageLatency(ms)=3D72.71] [READ AverageLatency(ms)=3D187.19]
>>> =C2=A020 sec: 68846 operations; 3814.14 current ops/sec; [UPDATE
>>> AverageLatency(ms)=3D65.36] [READ AverageLatency(ms)=3D191.75]
>>>
>>> We never measured more than ~6000 ops/sec. Are there ways to tune
>>> Cassandra that we are not aware of? We made some modification to the
>>> Cassandra 0.6.5 core for experimental reasons, so it's not easy to
>>> switch to 0.7x or 0.8x. However, if this might solve the scaling
>>> issues, we might consider to port our modifications to a newer
>>> Cassandra version...
>>>
>>> Thanks,
>>>
>>> Markus Klems
>>>
>>> Karlsruhe Institute of Technology, Germany
>>>
>