incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Markus Klems <markuskl...@gmail.com>
Subject Re: Benchmarking Cassandra with YCSB
Date Tue, 15 Feb 2011 22:20:39 GMT
Good point. When we looked at the EC2 nodes, we measured 120% CPU utilization or so. We interpreted
this as a false representation of CPU utilization on a multi-core machine. Our EC2 nodes have
8 virtual cores each.

Maybe Cassandra 0.6.5 is not so good with execution on multi-core systems?

On 15.02.2011, at 20:59, Thibaut Britz <thibaut.britz@trendiction.com> wrote:

> Cassandra is very CPU hungry so you might be hitting a CPU bottleneck.
> What's your CPU usage during these tests?
> 
> 
> On Tue, Feb 15, 2011 at 8:45 PM, Markus Klems <markus@klems.eu> wrote:
>> Hi there,
>> 
>> we are currently benchmarking a Cassandra 0.6.5 cluster with 3
>> High-Mem Quadruple Extra Large EC2 nodes
>> (http://aws.amazon.com/ec2/#instance) using Yahoo's YCSB tool
>> (replication factor is 3, random partitioner). We assigned 32 GB RAM
>> to the JVM and left 32 GB RAM for the Ubuntu Linux filesystem buffer.
>> We also set the user count to a very large number via ulimit -u
>> 999999.
>> 
>> Our goal is to achieve max throughput by increasing YCSB's threadcount
>> parameter (i.e. the number of parallel benchmarking client threads).
>> However, this does only improve Cassandra throughput for low numbers
>> of threads. If we move to higher threadcounts, throughput does not
>> increase and even  decreases. Do you have any idea why this is
>> happening and possibly suggestions how to scale throughput to much
>> higher numbers? Why is throughput hitting a wall, anyways? And where
>> does the latency/throughput tradeoff come from?
>> 
>> Here is our YCSB configuration:
>> recordcount=300000
>> operationcount=1000000
>> workload=com.yahoo.ycsb.workloads.CoreWorkload
>> readallfields=true
>> readproportion=0.5
>> updateproportion=0.5
>> scanproportion=0
>> insertproportion=0
>> threadcount= 500
>> target = 10000
>> hosts=EC2-1,EC2-2,EC2-3
>> requestdistribution=uniform
>> 
>> These are typical results for threadcount=1:
>> Loading workload...
>> Starting test.
>>  0 sec: 0 operations;
>>  10 sec: 11733 operations; 1168.28 current ops/sec; [UPDATE
>> AverageLatency(ms)=0.64] [READ AverageLatency(ms)=1.03]
>>  20 sec: 24246 operations; 1251.68 current ops/sec; [UPDATE
>> AverageLatency(ms)=0.48] [READ AverageLatency(ms)=1.11]
>> 
>> These are typical results for threadcount=10:
>> 10 sec: 30428 operations; 3029.77 current ops/sec; [UPDATE
>> AverageLatency(ms)=2.11] [READ AverageLatency(ms)=4.32]
>>  20 sec: 60838 operations; 3041.91 current ops/sec; [UPDATE
>> AverageLatency(ms)=2.15] [READ AverageLatency(ms)=4.37]
>> 
>> These are typical results for threadcount=100:
>> 10 sec: 29070 operations; 2895.42 current ops/sec; [UPDATE
>> AverageLatency(ms)=20.53] [READ AverageLatency(ms)=44.91]
>>  20 sec: 53621 operations; 2455.84 current ops/sec; [UPDATE
>> AverageLatency(ms)=23.11] [READ AverageLatency(ms)=55.39]
>> 
>> These are typical results for threadcount=500:
>> 10 sec: 30655 operations; 3053.59 current ops/sec; [UPDATE
>> AverageLatency(ms)=72.71] [READ AverageLatency(ms)=187.19]
>>  20 sec: 68846 operations; 3814.14 current ops/sec; [UPDATE
>> AverageLatency(ms)=65.36] [READ AverageLatency(ms)=191.75]
>> 
>> We never measured more than ~6000 ops/sec. Are there ways to tune
>> Cassandra that we are not aware of? We made some modification to the
>> Cassandra 0.6.5 core for experimental reasons, so it's not easy to
>> switch to 0.7x or 0.8x. However, if this might solve the scaling
>> issues, we might consider to port our modifications to a newer
>> Cassandra version...
>> 
>> Thanks,
>> 
>> Markus Klems
>> 
>> Karlsruhe Institute of Technology, Germany
>> 

Mime
View raw message