Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 08C1C9EBE for ; Wed, 18 Jul 2012 11:30:28 +0000 (UTC) Received: (qmail 88840 invoked by uid 500); 18 Jul 2012 11:30:25 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 88668 invoked by uid 500); 18 Jul 2012 11:30:24 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 88584 invoked by uid 99); 18 Jul 2012 11:30:23 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Jul 2012 11:30:23 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of mainalimanoj@gmail.com designates 209.85.160.44 as permitted sender) Received: from [209.85.160.44] (HELO mail-pb0-f44.google.com) (209.85.160.44) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Jul 2012 11:30:17 +0000 Received: by pbcwy7 with SMTP id wy7so2563090pbc.31 for ; Wed, 18 Jul 2012 04:29:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=KjUuH10uMmlWm69MRfbCt6rbYH8/IUx/5Xrp6f2E6Fc=; b=H/jXjPO7o5PeUUjgcdmcoMeYnn/veiRZaFrGFjQ2BweA65cDY9ETzTnqkgc6ruX92w nKawEfBAdTqyIY40Fe6J1+5/RkGeomIhm0tY3JIW0YCCJfH5IFIPA8TVEL/og28S8Gmm voPMO0HkT1I6ByGXw2diCo/tBK7F6IjET7PVk5eIU+PymxiKmiANg1KRB0GrTqobzXIY aayk1w3PoXuBpZXJSsxtVNV8WgN4d7iz9aGV5Jxl6X8/423glEURLV973uHg8rFRJ4lp f9pm8eGHe6fKZFYtkrbU5HBB9BsOgPbpHtSzsk9Mi1r77FB+dOoaQBL3UrmgnrotwqWQ 4jMw== MIME-Version: 1.0 Received: by 10.68.212.70 with SMTP id ni6mr6837261pbc.22.1342610995969; Wed, 18 Jul 2012 04:29:55 -0700 (PDT) Received: by 10.68.47.232 with HTTP; Wed, 18 Jul 2012 04:29:55 -0700 (PDT) In-Reply-To: References: <1C1DF018-9710-467B-9180-EC2CE58A6422@thelastpickle.com> Date: Wed, 18 Jul 2012 20:29:55 +0900 Message-ID: Subject: Re: Cassandra Evaluation/ Benchmarking: Throughput not scaling as expected neither latency showing good numbers From: Manoj Mainali To: "user@cassandra.apache.org" Content-Type: multipart/alternative; boundary=e89a8fb2085c7c7e8404c518fb02 --e89a8fb2085c7c7e8404c518fb02 Content-Type: text/plain; charset=ISO-8859-1 How kind of client are you using in YCSB? If you want to improve latency, try distributing the requests among nodes instead of stressing a single node, try host connection pooling instead of creating connection for each request. Check high level clients like hector or asyantax for use if you are not already using them. Some clients have ring aware request handling. You have a 3 nodes cluster and using a RF of three, that means all the node will get the data. What CL are you using for writes? Latency increases for strong CL. If you want to increase throughput, try increasing the number of clients. Of course, it doesnt mean that throughtput will always increase. My observation was that it will increase and after certain number of clients throughput decrease again. Regards, Manoj Mainali On Wednesday, July 18, 2012, Code Box wrote: > The cassandra stress tool gives me values around 2.5 milli seconds for > writing. The problem with the Cassandra Stress Tool is that it just gives > the average latency numbers and the average latency numbers that i am > getting are comparable in some cases. It is the 95 percentile and 99 > percentile numbers are the ones that are bad. So it means that the 95% of > requests are really bad and the rest 5% are really good that makes the > average go down. I want to make sure that the 95% and 99% values are in one > digit milli seconds. I want them to be single digit because i have seen > people getting those numbers. > > This is my conclusion till now with all the investigations:- > > Three node cluster with replication factor of 3 gets me around 10 ms 100% > writes with consistency equal to ONE. The reads are really bad and they are > around 65ms. > > I thought that network is the issue so i moved the client on a local > machine. Client on the local machine with one node cluster gives me again > good average write latencies but the 99%ile and 95%ile are bad. I am > getting around 10 ms for write and 25 ms for read. > > Network Bandwidth between the client and server is 1 Gigabit/second. I was > able to at the max generate 25 K requests. So it could be the client is the > bottleneck. I am using YCSB. May be i should change my client to some other. > > Throughput that i got from a client at the maximum local was 35K and > remote was 17K. > > > I can try these things now:- > > Use a different client and see how much numbers i get for 99% and 95%. I > am not sure if there is any client that gives me this detailed or i have to > write one of my own. > > Tweak some hard disk settings raid0 and xfs / ext4 and see if that helps. > > Could be a possibility that the cassandra 0.8 to 1.1 the 95% and 99% > numbers have gone down. The throughput numbers have also gone down. > > Is there any other client that i can use except the cassandra stress tool > and YCSB and what ever numbers i have got are they good ? > > > --Akshat Vig. > > > > > On Tue, Jul 17, 2012 at 9:22 PM, aaron morton wrote: > > I would benchmark a default installation, then start tweaking. That way > you can see if your changes result in improvements. > > To simplify things further try using the tools/stress utility in the > cassandra source distribution first. It's pretty simple to use. > > Add clients until you see the latency increase and tasks start to back up > in nodetool tpstats. If you see it report dropped messages it is over > loaded. > > Hope that helps. > > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 18/07/2012, at 4:48 AM, Code Box wrote: > > Thanks a lot for your reply guys. I was trying fsyn = batch and window > =0ms to see if the disk utilization is happening full on my drive. I > checked the numbers using iostat the numbers were around 60% and the CPU > usage was also not too high. > > Configuration of my Setup :- > > I have three m1.xlarge hosts each having 15 GB RAM and 4 CPU. It has 8 > EC2 Compute Units. > I have kept the replication factor equal to 3. The typical write size is 1 > KB. > > I tried adding different nodes each with 200 threads and the throughput > got split into two. If i do it from a single host with FSync Set to > Periodic and Window Size equal to 1000ms and using two nodes i am getting > these numbers :- > > > [OVERALL], Throughput(ops/sec), 4771 > [INSERT], AverageLatency(us), 18747 > [INSERT], MinLatency(us), 1470 > [INSERT], MaxLatency(us), 446413 > [INSERT], 95thPercentileLatency(ms), 55 > [INSERT], 99thPercentileLatency(ms), 167 > > [OVERALL], Throughput(ops/sec), 4678 > [INSERT], AverageLatency(us), 22015 > [INSERT], MinLatency(us), 1439 > [INSERT], MaxLatency(us), 466149 > [INSERT], 95thPercentileLatency(ms), 62 > [INSERT], 99thPercentileLatency(ms), 171 > > Is there something i am doing wrong in cassandra Setup ?? What is the bet > Setup for Cassandra to get high throughput and good write latency numbers ? > > > > On Tue, Jul 17, 2012 at 7:02 AM, Sylvain Lebresne > > --e89a8fb2085c7c7e8404c518fb02 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable How kind of client are you using in YCSB? If you want to improve latency, t= ry distributing the requests among nodes instead of stressing a single node= , try host connection pooling instead of creating connection for each reque= st. Check high level clients like hector or asyantax for use if you are not= already using them. Some clients have ring aware request handling.

You have a 3 nodes cluster and using a RF of three, that mea= ns all the node will get the data. What CL are you using for writes? Latenc= y increases for strong CL.=A0

If you want to incre= ase throughput, try increasing the number of clients. Of course, it doesnt = mean that throughtput will always increase. My observation was that it will= increase and after certain number of clients throughput decrease again.

Regards,
Manoj Mainali


On Wednesday, July 18, 2012, Code Box wrote:
The cassandra stress tool gives me values around 2.5 milli seconds for= writing. The problem with the Cassandra Stress Tool is that it just gives = the average latency numbers and the average latency numbers that i am getti= ng are comparable in some cases. It is the 95 percentile and 99 percentile = numbers are the ones that are bad. So it means that the 95% of requests are= really bad and the rest 5% are really good that makes the average go down.= I want to make sure that the 95% and 99% values are in one digit milli sec= onds. I want them to be single digit because i have seen people getting tho= se numbers.=A0

This is my conclusion till now with all the invest= igations:-

Three node cluster with replication fac= tor of 3 gets me around 10 ms 100% writes with consistency equal to ONE. Th= e reads are really bad and they are around 65ms.=A0

I thought that network is the issue so i moved the clie= nt on a local machine. Client on the local machine with one node cluster gi= ves me again good average write latencies but the 99%ile and 95%ile are bad= . I am getting around 10 ms for write and 25 ms for read.=A0

Network Bandwidth between the client and server is 1 Gi= gabit/second. I was able to at the max generate 25 K requests. So it could = be the client is the bottleneck. I am using YCSB. May be i should change my= client to some other.

Throughput that i got from a client at the maximum loca= l was 35K and remote was 17K.


I can= try these things now:-

Use a different client and= see how much numbers i get for 99% and 95%. I am not sure if there is any = client that gives me this detailed or i have to write one of my own.

Tweak some hard disk settings raid0 and xfs / ext4 and = see if that helps.

Could be a possibility that the= cassandra 0.8 to 1.1 the 95% and 99% numbers have gone down. =A0The throug= hput numbers have also gone down.

Is there any other client that i can use except the cas= sandra stress tool and YCSB =A0and what ever numbers i have got are they go= od ?


--Akshat Vig.




On Tue, Jul 17, 2012 at 9= :22 PM, aaron morton <aaron@thelastpickle.com&g= t; wrote:
I would benchmark a default in= stallation, then start tweaking. That way you can see if your changes resul= t in improvements.
=A0
To simplify things further try using the tools/stress utility= in the cassandra source distribution first. It's pretty simple to use.= =A0

Add clients until you see the latency increase= and tasks start to back up in nodetool tpstats. If you see it report dropp= ed messages it is over loaded.

Hope that helps.

<= div style=3D"word-wrap:break-word">
-----------------
Aaron Morton
Freelance Deve= loper
@aaronmorton

On 18/07/2012, at 4:48 AM, Code Box wrote:

Thanks a lot for your reply guys. I was trying fsyn =3D ba= tch and window =3D0ms to see if the disk utilization is happening full on m= y drive. I checked the =A0numbers using iostat the numbers were around 60% = and the CPU usage was also not too high.=A0

Configuration of my Setup :-

I have three= =A0m1.xlarge=A0hosts each = having 15 GB RAM and 4 CPU. It has I have kept the replication = factor equal to 3. The typical write size is 1 KB.=A0

I tried adding different nodes each wit= h 200 threads and the throughput got split into two. If i do it from a sing= le host with FSync Set to Periodic and Window Size equal to 1000ms and usin= g two nodes i am getting these numbers :-


[OVERALL], Throughput(ops/sec), 4771
[INSERT], AverageLatency(us)= , 18747
[INSERT], MinLatency(us), 1470
[INSERT], MaxLat= ency(us), 446413
[INSERT], 95thPercentileLatency(ms), 55
[INSERT], 99thPercentileLatency(ms), 167

[OVE= RALL], Throughput(ops/sec), 4678
[INSERT], AverageLatency(us= ), 22015
[INSERT], MinLatency(us), 1439
[INSERT], MaxLa= tency(us), 466149
[INSERT], 95thPercentileLatency(ms), 62
[INSERT], 99thPercen= tileLatency(ms), 171

Is there something i am= doing wrong in cassandra Setup ?? What is the bet Setup for Cassandra to g= et high throughput and good write latency numbers ?



On Tue, Jul 1= 7, 2012 at 7:02 AM, Sylvain Lebresne <sylvain@datas= tax.com>
--e89a8fb2085c7c7e8404c518fb02--