cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Code Box <codeith...@gmail.com>
Subject Re: Cassandra Evaluation/ Benchmarking: Throughput not scaling as expected neither latency showing good numbers
Date Sat, 21 Jul 2012 06:01:50 GMT
Thanks for the suggestion. I was able to get better results tuning the GC
settings but still not that great. I was seeing reading the netflix blog
for the settings they have done and they have posted on blog. But i could
not get close to what they are saying.

http://techblog.netflix.com/2012/07/benchmarking-high-performance-io-with.html


On Thu, Jul 19, 2012 at 9:45 PM, aaron morton <aaron@thelastpickle.com>wrote:

> Three node cluster with replication factor of 3 gets me around 10 ms 100%
>> writes with consistency equal to ONE. The reads are really bad and they are
>> around 65ms.
>>
> Using CL ONE in that situation, with a test that runs in a tight loop, can
> result in the clients overloading the cluster.
>
> Every node is a replica, so a write at CL ONE only has to wait for the
> local not to ACK. It will then return to the client before the remote nodes
> ACK, which means the client can send another request very quickly. In
> normal operation this may not be an issue, but load tests that run in a
> tight loop do not generate normal traffic.
>
> A better approach is to work at QUOURM so that network latency slows down
> individual client threads. Or generating the traffic using the Poisson
> distribution. The new load test from twitter uses that
> https://github.com/twitter/iago/ or you can use numpy for python.
>
> Cheers
>
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 18/07/2012, at 11:29 PM, Manoj Mainali wrote:
>
> How kind of client are you using in YCSB? If you want to improve latency,
> try distributing the requests among nodes instead of stressing a single
> node, try host connection pooling instead of creating connection for each
> request. Check high level clients like hector or asyantax for use if you
> are not already using them. Some clients have ring aware request handling.
>
> You have a 3 nodes cluster and using a RF of three, that means all the
> node will get the data. What CL are you using for writes? Latency increases
> for strong CL.
>
> If you want to increase throughput, try increasing the number of clients.
> Of course, it doesnt mean that throughtput will always increase. My
> observation was that it will increase and after certain number of clients
> throughput decrease again.
>
> Regards,
> Manoj Mainali
>
>
> On Wednesday, July 18, 2012, Code Box wrote:
>
>> The cassandra stress tool gives me values around 2.5 milli seconds for
>> writing. The problem with the Cassandra Stress Tool is that it just gives
>> the average latency numbers and the average latency numbers that i am
>> getting are comparable in some cases. It is the 95 percentile and 99
>> percentile numbers are the ones that are bad. So it means that the 95% of
>> requests are really bad and the rest 5% are really good that makes the
>> average go down. I want to make sure that the 95% and 99% values are in one
>> digit milli seconds. I want them to be single digit because i have seen
>> people getting those numbers.
>>
>> This is my conclusion till now with all the investigations:-
>>
>> Three node cluster with replication factor of 3 gets me around 10 ms 100%
>> writes with consistency equal to ONE. The reads are really bad and they are
>> around 65ms.
>>
>> I thought that network is the issue so i moved the client on a local
>> machine. Client on the local machine with one node cluster gives me again
>> good average write latencies but the 99%ile and 95%ile are bad. I am
>> getting around 10 ms for write and 25 ms for read.
>>
>> Network Bandwidth between the client and server is 1 Gigabit/second. I
>> was able to at the max generate 25 K requests. So it could be the client is
>> the bottleneck. I am using YCSB. May be i should change my client to some
>> other.
>>
>> Throughput that i got from a client at the maximum local was 35K and
>> remote was 17K.
>>
>>
>> I can try these things now:-
>>
>> Use a different client and see how much numbers i get for 99% and 95%. I
>> am not sure if there is any client that gives me this detailed or i have to
>> write one of my own.
>>
>> Tweak some hard disk settings raid0 and xfs / ext4 and see if that helps.
>>
>> Could be a possibility that the cassandra 0.8 to 1.1 the 95% and 99%
>> numbers have gone down.  The throughput numbers have also gone down.
>>
>> Is there any other client that i can use except the cassandra stress tool
>> and YCSB  and what ever numbers i have got are they good ?
>>
>>
>> --Akshat Vig.
>>
>>
>>
>>
>> On Tue, Jul 17, 2012 at 9:22 PM, aaron morton <aaron@thelastpickle.com>wrote:
>>
>> I would benchmark a default installation, then start tweaking. That way
>> you can see if your changes result in improvements.
>>
>> To simplify things further try using the tools/stress utility in the
>> cassandra source distribution first. It's pretty simple to use.
>>
>> Add clients until you see the latency increase and tasks start to back up
>> in nodetool tpstats. If you see it report dropped messages it is over
>> loaded.
>>
>> Hope that helps.
>>
>>   -----------------
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 18/07/2012, at 4:48 AM, Code Box wrote:
>>
>> Thanks a lot for your reply guys. I was trying fsyn = batch and window
>> =0ms to see if the disk utilization is happening full on my drive. I
>> checked the  numbers using iostat the numbers were around 60% and the CPU
>> usage was also not too high.
>>
>> Configuration of my Setup :-
>>
>> I have three m1.xlarge hosts each having 15 GB RAM and 4 CPU. It has 8
>> EC2 Compute Units.
>> I have kept the replication factor equal to 3. The typical write size is
>> 1 KB.
>>
>> I tried adding different nodes each with 200 threads and the throughput
>> got split into two. If i do it from a single host with FSync Set to
>> Periodic and Window Size equal to 1000ms and using two nodes i am getting
>> these numbers :-
>>
>>
>> [OVERALL], Throughput(ops/sec), 4771
>> [INSERT], AverageLatency(us), 18747
>> [INSERT], MinLatency(us), 1470
>> [INSERT], MaxLatency(us), 446413
>> [INSERT], 95thPercentileLatency(ms), 55
>> [INSERT], 99thPercentileLatency(ms), 167
>>
>> [OVERALL], Throughput(ops/sec), 4678
>> [INSERT], AverageLatency(us), 22015
>> [INSERT], MinLatency(us), 1439
>> [INSERT], MaxLatency(us), 466149
>> [INSERT], 95thPercentileLatency(ms), 62
>> [INSERT], 99thPercentileLatency(ms), 171
>>
>> Is there something i am doing wrong in cassandra Setup ?? What is the bet
>> Setup for Cassandra to get high throughput and good write latency numbers ?
>>
>>
>>
>> On Tue, Jul 17, 2012 at 7:02 AM, Sylvain Lebresne <sylvain@datastax.com>
>>
>>
>

Mime
View raw message