cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Cassandra Evaluation/ Benchmarking: Throughput not scaling as expected neither latency showing good numbers
Date Wed, 18 Jul 2012 04:22:24 GMT
I would benchmark a default installation, then start tweaking. That way you can see if your
changes result in improvements.
 
To simplify things further try using the tools/stress utility in the cassandra source distribution
first. It's pretty simple to use. 

Add clients until you see the latency increase and tasks start to back up in nodetool tpstats.
If you see it report dropped messages it is over loaded.

Hope that helps.

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 18/07/2012, at 4:48 AM, Code Box wrote:

> Thanks a lot for your reply guys. I was trying fsyn = batch and window =0ms to see if
the disk utilization is happening full on my drive. I checked the  numbers using iostat the
numbers were around 60% and the CPU usage was also not too high. 
> 
> Configuration of my Setup :-
> 
> I have three m1.xlarge hosts each having 15 GB RAM and 4 CPU. It has 8 EC2 Compute Units.
> I have kept the replication factor equal to 3. The typical write size is 1 KB. 
> 
> I tried adding different nodes each with 200 threads and the throughput got split into
two. If i do it from a single host with FSync Set to Periodic and Window Size equal to 1000ms
and using two nodes i am getting these numbers :-
> 
> 
> [OVERALL], Throughput(ops/sec), 4771
> [INSERT], AverageLatency(us), 18747
> [INSERT], MinLatency(us), 1470
> [INSERT], MaxLatency(us), 446413
> [INSERT], 95thPercentileLatency(ms), 55
> [INSERT], 99thPercentileLatency(ms), 167
> 
> [OVERALL], Throughput(ops/sec), 4678
> [INSERT], AverageLatency(us), 22015
> [INSERT], MinLatency(us), 1439
> [INSERT], MaxLatency(us), 466149
> [INSERT], 95thPercentileLatency(ms), 62
> [INSERT], 99thPercentileLatency(ms), 171
> 
> Is there something i am doing wrong in cassandra Setup ?? What is the bet Setup for Cassandra
to get high throughput and good write latency numbers ?
> 
> 
> 
> On Tue, Jul 17, 2012 at 7:02 AM, Sylvain Lebresne <sylvain@datastax.com> wrote:
> FSync = Batch and Window = 0ms is expected to give relatively crappy result. It means
C* will fsync on disk pretty much all write. This is an overly safe setting and no database
with that kind of setting will perform correctly because you're far too much bound by the
hard drive.
> 
> If you want strong local durability, use Batch (so that C* never ack a non-fsynced write)
but keep a bigger window. And in any case, Periodic will give you better results and provided
you use a replication factor > 1, it is good enough in 99% of the case.
> 
> As for the exact numbers, you didn't even say what kind of instance you are using, nor
the replication factor, nor the typical size of each write, so it's hard to tell you if it
seems reasonable or not.
> 
> As for the scalability, as horschi said, it's about adding nodes, not adding clients.
> 
> --
> Sylvain
>  
> 
> On Tue, Jul 17, 2012 at 3:43 PM, horschi <horschi@gmail.com> wrote:
> When they say "linear scalibility" they mean "throughput scales with the amount of machines
in your cluster".
> 
> Try adding more machines to your cluster and measure the thoughput. I'm pretty sure you'll
see linear scalibility.
> 
> regards,
> Christian
> 
> 
> 
> On Tue, Jul 17, 2012 at 6:13 AM, Code Box <codeithere@gmail.com> wrote:
> I am doing Cassandra Benchmarking using YCSB for evaluating the best performance for
my application which will be both read and write intensive. I have set up a three cluster
environment on EC2 and i am using YCSB in the same availability region as a client. I have
tried various combinations of tuning cassandra parameters like FSync ( Setting to batch and
periodic ), Increasing the number of rpc_threads, increasing number of concurrent reads and
concurrent writes, write consistency one and Quorum i am not getting very great results and
also i do not see a linear graph in terms of scalability that is if i increase the number
of clients i do not see an increase in the throughput.
> 
> Here are some sample numbers that i got :-
> 
> Test 1:-  Write Consistency set to Quorum Write Proportion = 100%. FSync = Batch and
Window = 0ms
> 
> Threads	Throughput ( write per sec )	Avg Latency (ms)	TP95(ms)	 TP99(ms)	Min(ms)	Max(ms)

> 
> 
> 10	2149	3.198	4	 5	1.499	291	   
> 100	 4070	23.8	28	70	2.2	260	   
> 200	4151	 45.96	57	130	 1.7	1242	    
> 300	4197	64.68	 115	422	2.09	 216            	
> 
> 
> If you look at the numbers the number of threads do not increase the throughput. Also
the latency values are not that great. I am using fsync set to batch and with 0 ms window.
> 
> Test 2:-  Write Consistency set to Quorum Write Proportion = 100%. FSync = Periodic and
Window = 1000 ms
> 
> 1	803	 1.237	1	2	 1.012	312.9	Q
> 100	15944	 5.343	9	25	 1.21	579.1	Q
> 200	19630	9.047	 19	70	1.17	 1851	Q
> 
> Are these numbers expected numbers or does Cassandra perform better ? Am i missing something
?
> 
> 
> 


Mime
View raw message