Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 540A69FBA for ; Wed, 18 Jul 2012 04:23:01 +0000 (UTC) Received: (qmail 78117 invoked by uid 500); 18 Jul 2012 04:22:59 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 77966 invoked by uid 500); 18 Jul 2012 04:22:58 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 77949 invoked by uid 99); 18 Jul 2012 04:22:58 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Jul 2012 04:22:58 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a94.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Jul 2012 04:22:50 +0000 Received: from homiemail-a94.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a94.g.dreamhost.com (Postfix) with ESMTP id 7D99138A059 for ; Tue, 17 Jul 2012 21:22:27 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; q=dns; s=thelastpickle.com; b=aPd+b6fOC7 uyNz4c+E2TGh7Tw1q29+TUfjxOgsrPprvRxrUogsQDUwve8NFgZN3Grlx3Rat6By +dPYLueKPbn3LPtu4afbmm/7BFL7RMXf75lI93u7qtmCN3fguk3jpst1IBJe81dH twiRRX0sQvvZfWpqem85r7pksVdg57ziY= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; s=thelastpickle.com; bh=y9uWnfdCcz4tAIra DNhiVJ3gk5w=; b=gaanIiNS4xCC6cYFb3O8x0uD12hCTw4cMvXre7Eeq9rankv/ 9qWcnp5JFUE8Zt7SLzDURdG0+XEgwz4drBBYojOt3TifrjfeZAqYCHD9N4LugOkI kF9S2jtZjOsmlZ3D9IctEN3msPnDBcneR86unOYljOA0YSS5JT3IR6C40OM= Received: from [192.168.2.189] (unknown [116.90.132.105]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a94.g.dreamhost.com (Postfix) with ESMTPSA id D3D1538A00B for ; Tue, 17 Jul 2012 21:22:26 -0700 (PDT) From: aaron morton Mime-Version: 1.0 (Apple Message framework v1278) Content-Type: multipart/alternative; boundary="Apple-Mail=_C208D9B1-0EEE-4083-93A9-77CA9C174C78" Subject: Re: Cassandra Evaluation/ Benchmarking: Throughput not scaling as expected neither latency showing good numbers Date: Wed, 18 Jul 2012 16:22:24 +1200 In-Reply-To: To: user@cassandra.apache.org References: Message-Id: <1C1DF018-9710-467B-9180-EC2CE58A6422@thelastpickle.com> X-Mailer: Apple Mail (2.1278) --Apple-Mail=_C208D9B1-0EEE-4083-93A9-77CA9C174C78 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 I would benchmark a default installation, then start tweaking. That way = you can see if your changes result in improvements. =20 To simplify things further try using the tools/stress utility in the = cassandra source distribution first. It's pretty simple to use.=20 Add clients until you see the latency increase and tasks start to back = up in nodetool tpstats. If you see it report dropped messages it is over = loaded. Hope that helps. ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 18/07/2012, at 4:48 AM, Code Box wrote: > Thanks a lot for your reply guys. I was trying fsyn =3D batch and = window =3D0ms to see if the disk utilization is happening full on my = drive. I checked the numbers using iostat the numbers were around 60% = and the CPU usage was also not too high.=20 >=20 > Configuration of my Setup :- >=20 > I have three m1.xlarge hosts each having 15 GB RAM and 4 CPU. It has 8 = EC2 Compute Units. > I have kept the replication factor equal to 3. The typical write size = is 1 KB.=20 >=20 > I tried adding different nodes each with 200 threads and the = throughput got split into two. If i do it from a single host with FSync = Set to Periodic and Window Size equal to 1000ms and using two nodes i am = getting these numbers :- >=20 >=20 > [OVERALL], Throughput(ops/sec), 4771 > [INSERT], AverageLatency(us), 18747 > [INSERT], MinLatency(us), 1470 > [INSERT], MaxLatency(us), 446413 > [INSERT], 95thPercentileLatency(ms), 55 > [INSERT], 99thPercentileLatency(ms), 167 >=20 > [OVERALL], Throughput(ops/sec), 4678 > [INSERT], AverageLatency(us), 22015 > [INSERT], MinLatency(us), 1439 > [INSERT], MaxLatency(us), 466149 > [INSERT], 95thPercentileLatency(ms), 62 > [INSERT], 99thPercentileLatency(ms), 171 >=20 > Is there something i am doing wrong in cassandra Setup ?? What is the = bet Setup for Cassandra to get high throughput and good write latency = numbers ? >=20 >=20 >=20 > On Tue, Jul 17, 2012 at 7:02 AM, Sylvain Lebresne = wrote: > FSync =3D Batch and Window =3D 0ms is expected to give relatively = crappy result. It means C* will fsync on disk pretty much all write. = This is an overly safe setting and no database with that kind of setting = will perform correctly because you're far too much bound by the hard = drive. >=20 > If you want strong local durability, use Batch (so that C* never ack a = non-fsynced write) but keep a bigger window. And in any case, Periodic = will give you better results and provided you use a replication factor > = 1, it is good enough in 99% of the case. >=20 > As for the exact numbers, you didn't even say what kind of instance = you are using, nor the replication factor, nor the typical size of each = write, so it's hard to tell you if it seems reasonable or not. >=20 > As for the scalability, as horschi said, it's about adding nodes, not = adding clients. >=20 > -- > Sylvain > =20 >=20 > On Tue, Jul 17, 2012 at 3:43 PM, horschi wrote: > When they say "linear scalibility" they mean "throughput scales with = the amount of machines in your cluster". >=20 > Try adding more machines to your cluster and measure the thoughput. = I'm pretty sure you'll see linear scalibility. >=20 > regards, > Christian >=20 >=20 >=20 > On Tue, Jul 17, 2012 at 6:13 AM, Code Box = wrote: > I am doing Cassandra Benchmarking using YCSB for evaluating the best = performance for my application which will be both read and write = intensive. I have set up a three cluster environment on EC2 and i am = using YCSB in the same availability region as a client. I have tried = various combinations of tuning cassandra parameters like FSync ( Setting = to batch and periodic ), Increasing the number of rpc_threads, = increasing number of concurrent reads and concurrent writes, write = consistency one and Quorum i am not getting very great results and also = i do not see a linear graph in terms of scalability that is if i = increase the number of clients i do not see an increase in the = throughput. >=20 > Here are some sample numbers that i got :- >=20 > Test 1:- Write Consistency set to Quorum Write Proportion =3D 100%. = FSync =3D Batch and Window =3D 0ms >=20 > Threads Throughput ( write per sec ) Avg Latency (ms) = TP95(ms) TP99(ms) Min(ms) Max(ms)=09 >=20 >=20 > 10 2149 3.198 4 5 1.499 291 =20 > 100 4070 23.8 28 70 2.2 260 =20 > 200 4151 45.96 57 130 1.7 1242 =20 > 300 4197 64.68 115 422 2.09 216 =09 >=20 >=20 > If you look at the numbers the number of threads do not increase the = throughput. Also the latency values are not that great. I am using fsync = set to batch and with 0 ms window. >=20 > Test 2:- Write Consistency set to Quorum Write Proportion =3D 100%. = FSync =3D Periodic and Window =3D 1000 ms >=20 > 1 803 1.237 1 2 1.012 312.9 Q > 100 15944 5.343 9 25 1.21 579.1 Q > 200 19630 9.047 19 70 1.17 1851 Q >=20 > Are these numbers expected numbers or does Cassandra perform better ? = Am i missing something ? >=20 >=20 >=20 --Apple-Mail=_C208D9B1-0EEE-4083-93A9-77CA9C174C78 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=iso-8859-1 I = would benchmark a default installation, then start tweaking. That way = you can see if your changes result in = improvements.
 
To simplify things further try using = the tools/stress utility in the cassandra source distribution first. = It's pretty simple to use. 

Add clients = until you see the latency increase and tasks start to back up in = nodetool tpstats. If you see it report dropped messages it is over = loaded.

Hope that helps.

http://www.thelastpickle.com

On 18/07/2012, at 4:48 AM, Code Box wrote:

Thanks a = lot for your reply guys. I was trying fsyn =3D batch and window =3D0ms = to see if the disk utilization is happening full on my drive. I checked = the  numbers using iostat the numbers were around 60% and the CPU = usage was also not too high. 

Configuration of my Setup :-

I have = three m1.= xlarge hosts each having 15 GB RAM and 4 CPU. It has 8 = EC2 Compute Units.
I = have kept the replication factor equal to 3. The typical write size is 1 = KB. 
I = tried adding different nodes each with 200 threads and the throughput = got split into two. If i do it from a single host with FSync Set to = Periodic and Window Size equal to 1000ms and using two nodes i am = getting these numbers :-

[OVERALL], Throughput(ops/sec), 4771
[INSERT], = AverageLatency(us), 18747
[INSERT], MinLatency(us), = 1470
[INSERT], MaxLatency(us), 446413
[INSERT], = 95thPercentileLatency(ms), 55
[INSERT], 99thPercentileLatency(ms), = 167

[OVERALL], Throughput(ops/sec), = 4678
[INSERT], AverageLatency(us), = 22015
[INSERT], MinLatency(us), 1439
[INSERT], = MaxLatency(us), 466149
[INSERT], 95thPercentileLatency(ms), 62
[INSERT], = 99thPercentileLatency(ms), 171

Is there = something i am doing wrong in cassandra Setup ?? What is the bet Setup = for Cassandra to get high throughput and good write latency numbers = ?



On Tue, Jul 17, 2012 at 7:02 AM, Sylvain Lebresne = <sylvain@datastax.com> wrote:
FSync =3D Batch and = Window =3D 0ms is expected to give relatively crappy result. It means C* = will fsync on disk pretty much all write. This is an overly safe = setting and no database with that kind of setting will perform correctly = because you're far too much bound by the hard drive.

If you want strong local durability, use Batch (so that = C* never ack a non-fsynced write) but keep a bigger window. And in any = case, Periodic will give you better results and provided you use a = replication factor > 1, it is good enough in 99% of the case.

As for the exact numbers, you didn't even say what = kind of instance you are using, nor the replication factor, nor the = typical size of each write, so it's hard to tell you if it seems = reasonable or not.

As for the scalability, as horschi said, it's about = adding nodes, not adding = clients.

--
Sylvain
 

On Tue, Jul 17, 2012 at 3:43 PM, horschi <horschi@gmail.com> wrote:
When they say "linear = scalibility" they mean "throughput scales with the amount of machines in = your cluster".

Try adding more machines to your cluster and measure the thoughput. = I'm pretty sure you'll see linear scalibility.

regards,
Christian



On Tue, Jul 17, 2012 at 6:13 AM, Code = Box <codeithere@gmail.com> = wrote:

I am doing Cassandra Benchmarking using = YCSB for evaluating the best performance for my application which will = be both read and write intensive. I have set up a three cluster = environment on EC2 and i am using YCSB in the same availability region = as a client. I have tried various combinations of tuning cassandra = parameters like FSync ( Setting to batch and periodic ), Increasing the = number of rpc_threads, increasing number of concurrent reads and = concurrent writes, write consistency one and Quorum i am not getting = very great results and also i do not see a linear graph in terms of = scalability that is if i increase the number of clients i do not see an = increase in the throughput.

Here are = some sample numbers that i got :-

Test 1:- =  Write Consistency set to Quorum Write Proportion =3D 100%. FSync =3D= Batch and Window =3D 0ms

ThreadsThroughput ( write per sec ) Avg Latency (ms)TP95(ms)= TP99(ms)Min(ms)Max(ms)




If you look at the numbers the number of threads do not = increase the throughput. Also the latency values are not that great. I = am using fsync set to batch and with 0 ms window.

Test 2:-  Write Consistency = set to Quorum Write Proportion =3D 100%. FSync =3D Periodic and Window =3D= 1000 ms

102149 3.1984 51.499291   
100 4070 23.82870 2.2260  =  
2004151 45.9657130 1.71242  =   
300419764.68 1154222.09 216            

Are these numbers expected = numbers or does Cassandra perform better ? Am i missing something = ?




= --Apple-Mail=_C208D9B1-0EEE-4083-93A9-77CA9C174C78--
1803 1.23712 1.012312.9Q
10015944 5.343925 1.21579.1Q
200196309.047 19701.17 1851Q