Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C5EEF10D03 for ; Mon, 8 Dec 2014 10:58:37 +0000 (UTC) Received: (qmail 94906 invoked by uid 500); 8 Dec 2014 10:58:33 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 94865 invoked by uid 500); 8 Dec 2014 10:58:33 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 94855 invoked by uid 99); 8 Dec 2014 10:58:33 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Dec 2014 10:58:33 +0000 X-ASF-Spam-Status: No, hits=1.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of kongjialin92@gmail.com designates 209.85.223.171 as permitted sender) Received: from [209.85.223.171] (HELO mail-ie0-f171.google.com) (209.85.223.171) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Dec 2014 10:58:06 +0000 Received: by mail-ie0-f171.google.com with SMTP id rl12so4247975iec.16 for ; Mon, 08 Dec 2014 02:57:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=fCN67fedXlsl34+01SDk1zqRXQ3LvwhQAXkxqsMhuWA=; b=o/26iTQghVQTjs/+UDsJ1SlrteKdMnyh/pNp/aRqoMbaA21xzFP4vki5yUF+OskGOK ZV6/Gy+hS5EOO8JNlA4+odlC7ofnjtuysIQrATN8y8nl0weKgDlczGqkjvOalGCPPEDL gxgBO18kLz5m+chcyvuHhzzZIBrt41CD/20FYDPK3LQgIbVDxSByJt+DkqsXFjJRbDdf 0qRl5uP7vHe4r1zyK+ew2rxe384uXmLHsNYqbHGxb4S94o8h9i7Nc7bH49GfGN4Rq1gf LjmAcKzWl14KF7SBRuNw4dglP5vwkL65/P5p4FwbzHxAPp7XgDP1I1M87hxb/Cs+rR06 Dufg== MIME-Version: 1.0 X-Received: by 10.42.112.136 with SMTP id y8mr24972235icp.86.1418036240215; Mon, 08 Dec 2014 02:57:20 -0800 (PST) Received: by 10.64.142.100 with HTTP; Mon, 8 Dec 2014 02:57:20 -0800 (PST) In-Reply-To: References: <016301d0115e$1565de60$40319b20$@gmail.com> Date: Mon, 8 Dec 2014 18:57:20 +0800 Message-ID: Subject: Re: Cassandra Doesn't Get Linear Performance Increment in Stress Test on Amazon EC2 From: =?UTF-8?B?5a2U5ZiJ5p6X?= To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001a11c316a260743b0509b24a6b X-Virus-Checked: Checked by ClamAV on apache.org --001a11c316a260743b0509b24a6b Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Thanks Chris. I run a *client on a separate* AWS *instance from* the Cassandra cluster servers. At the client side, I create 40 or 50 threads for sending requests to each Cassandra node. I create one thrift client for each of the threads. And at the beginning, all the created thrift clients connect to the corresponding Cassandra nodes and keep connecting during the whole process(I did not close all the transports until the end of the test process). So I use very simple load balancing, since the same number of thrift clients connect to each node. And my source code is here: https://github.com/kongjialin/Cassandra/blob/master/cassandra_client.cpp It= 's very nice of you to help me improve my code. As I increase the number of threads, the latency gets longer. I'm using C++, so if I want to use native binary + prepared statements, the only way is to use C++ driver? Thanks very much. 2014-12-08 12:51 GMT+08:00 Chris Lohfink : > I think your client could use improvements. How many threads do you have > running in your test? With a thrift call like that you only can do one > request at a time per connection. For example, assuming C* takes 0ms, a > 10ms network latency/driver overhead will mean 20ms RTT and a max > throughput of ~50 QPS per thread (native binary doesn't behave like this)= . > Are you running client on its own system or shared with a node? how are > you load balancing your requests? Source code would help since theres a > lot that can become a bottleneck. > > Generally you will see a bit of a dip in latency from N=3DRF=3D1 and N=3D= 2, RF=3D2 > etc since there are optimizations on the coordinator node when it doesn't > need to send the request to the replicas. The impact of the network > overhead decreases in significance as cluster grows. Typically; latency > wise, RF=3DN=3D1 is going to be fastest possible for smaller loads (ie wh= en a > client cannot fully saturate a single node). > > Main thing to expect is that latency will plateau and remain fairly > constant as load/nodes increase while throughput potential will linearly > (empirically at least) increase. > > You should really attempt it with the native binary + prepared statements= , > running cql over thrift is far from optimal. I would recommend using the > cassandra-stress tool if you want to stress test Cassandra (and not your > code) > http://www.datastax.com/dev/blog/improved-cassandra-2-1-stress-tool-bench= mark-any-schema > > =3D=3D=3D > Chris Lohfink > > On Sun, Dec 7, 2014 at 9:48 PM, =E5=AD=94=E5=98=89=E6=9E=97 wrote: > >> Hi Eric, >> Thank you very much for your reply! >> Do you mean that I should clear my table after each run? Indeed, I can >> see several times of compaction during my test, but could only a few tim= es >> compaction affect the performance that much? Also, I can see from the >> OpsCenter some ParNew GC happen but no CMS GC happen. >> >> I run my test on EC2 cluster, I think the network could be of high speed >> with in it. Each Cassandra server has 4 units CPU, 15 GiB memory and 80 = SSD >> storage, which is of m3.xlarge type. >> >> As for latency, which latency should I care about most? p(99) or p(999)? >> I want to get the max QPS under a certain limited latency. >> >> I know my testing scenario are not the common case in production, I just >> want to know how much burden my cluster can bear under stress. >> >> So, how did you test your cluster that can get 86k writes/sec? How many >> requests did you send to your cluster? Was it also 1 million? Did you al= so >> use OpsCenter to monitor the real time performance? I also wonder why th= e >> write and read QPS OpsCenter provide are much lower than what I calculat= e. >> Could you please describe in detail about your test deployment? >> >> Thank you very much, >> Joy >> >> 2014-12-07 23:55 GMT+08:00 Eric Stevens : >> >>> Hi Joy, >>> >>> Are you resetting your data after each test run? I wonder if your test= s >>> are actually causing you to fall behind on data grooming tasks such as >>> compaction, and so performance suffers for your later tests. >>> >>> There are *so many* factors which can affect performance, without >>> reviewing test methodology in great detail, it's really hard to say whe= ther >>> there are flaws which might uncover an antipattern cause atypical numbe= r of >>> cache hits or misses, and so forth. You may also be producing gc pressu= re >>> in the write path, and so forth. >>> >>> I *can* say that 28k writes per second looks just a little low, but it >>> depends a lot on your network, hardware, and write patterns (eg, data >>> size). For a little performance test suite I wrote, with parallel batc= hed >>> writes, on a 3 node rf=3D3 cluster test cluster, I got about 86k writes= per >>> second. >>> >>> Also focusing exclusively on max latency is going to cause you some >>> troubles especially in the case of magnetic media as you're using. Bet= ween >>> ill-timed GC and inconsistent performance characteristics from magnetic >>> media, your max numbers will often look significantly worse than your p= (99) >>> or p(999) numbers. >>> >>> All this said, one node will often look better than several nodes for >>> certain patterns because it completely eliminates proxy (coordinator) w= rite >>> times. All writes are local writes. It's an over-simple case that doe= sn't >>> reflect any practical production use of Cassandra, so it's probably not >>> worth even including in your tests. I would recommend start at 3 nodes >>> rf=3D3, and compare against 6 nodes rf=3D6. Make sure you're staying o= n top of >>> compaction and aren't seeing garbage collections in the logs (either of >>> those will be polluting your results with variability you can't account= for >>> with small sample sizes of ~1 million). >>> >>> If you expect to sustain write volumes like this, you'll find these >>> clusters are sized too small (on that hardware you won't keep up with >>> compaction), and your tests are again testing scenarios you wouldn't >>> actually see in production. >>> >>> On Sat Dec 06 2014 at 7:09:18 AM kong wrote: >>> >>>> Hi, >>>> >>>> I am doing stress test on Datastax Cassandra Community 2.1.2, not usin= g >>>> the provided stress test tool, but use my own stress-test client code >>>> instead(I write some C++ stress test code). My Cassandra cluster is >>>> deployed on Amazon EC2, using the provided Datastax Community AMI( HVM >>>> instances ) in the Datastax document, and I am not using EBS, just usi= ng >>>> the ephemeral storage by default. The EC2 type of Cassandra servers ar= e >>>> m3.xlarge. I use another EC2 instance for my stress test client, which= is >>>> of type r3.8xlarge. Both the Cassandra sever nodes and stress test cli= ent >>>> node are in us-east. I test the Cassandra cluster which is made up of = 1 >>>> node, 2 nodes, and 4 nodes separately. I just do INSERT test and SELEC= T >>>> test separately, but the performance doesn=E2=80=99t get linear increm= ent when new >>>> nodes are added. Also I get some weird results. My test results are as >>>> follows(*I do 1 million operations and I try to get the best QPS when >>>> the max latency is no more than 200ms, and the latencies are measured = from >>>> the client side. The QPS is calculated by total_operations/total_time)= .* >>>> >>>> >>>> >>>> *INSERT(write):* >>>> >>>> Node count >>>> >>>> Replication factor >>>> >>>> QPS >>>> >>>> Average latency(ms) >>>> >>>> Min latency(ms) >>>> >>>> .95 latency(ms) >>>> >>>> .99 latency(ms) >>>> >>>> .999 latency(ms) >>>> >>>> Max latency(ms) >>>> >>>> 1 >>>> >>>> 1 >>>> >>>> 18687 >>>> >>>> 2.08 >>>> >>>> 1.48 >>>> >>>> 2.95 >>>> >>>> 5.74 >>>> >>>> 52.8 >>>> >>>> 205.4 >>>> >>>> 2 >>>> >>>> 1 >>>> >>>> 20793 >>>> >>>> 3.15 >>>> >>>> 0.84 >>>> >>>> 7.71 >>>> >>>> 41.35 >>>> >>>> 88.7 >>>> >>>> 232.7 >>>> >>>> 2 >>>> >>>> 2 >>>> >>>> 22498 >>>> >>>> 3.37 >>>> >>>> 0.86 >>>> >>>> 6.04 >>>> >>>> 36.1 >>>> >>>> 221.5 >>>> >>>> 649.3 >>>> >>>> 4 >>>> >>>> 1 >>>> >>>> 28348 >>>> >>>> 4.38 >>>> >>>> 0.85 >>>> >>>> 8.19 >>>> >>>> 64.51 >>>> >>>> 169.4 >>>> >>>> 251.9 >>>> >>>> 4 >>>> >>>> 3 >>>> >>>> 28631 >>>> >>>> 5.22 >>>> >>>> 0.87 >>>> >>>> 18.68 >>>> >>>> 68.35 >>>> >>>> 167.2 >>>> >>>> 288 >>>> >>>> >>>> >>>> *SELECT(read):* >>>> >>>> Node count >>>> >>>> Replication factor >>>> >>>> QPS >>>> >>>> Average latency(ms) >>>> >>>> Min latency(ms) >>>> >>>> .95 latency(ms) >>>> >>>> .99 latency(ms) >>>> >>>> .999 latency(ms) >>>> >>>> Max latency(ms) >>>> >>>> 1 >>>> >>>> 1 >>>> >>>> 24498 >>>> >>>> 4.01 >>>> >>>> 1.51 >>>> >>>> 7.6 >>>> >>>> 12.51 >>>> >>>> 31.5 >>>> >>>> 129.6 >>>> >>>> 2 >>>> >>>> 1 >>>> >>>> 28219 >>>> >>>> 3.38 >>>> >>>> 0.85 >>>> >>>> 9.5 >>>> >>>> 17.71 >>>> >>>> 39.2 >>>> >>>> 152.2 >>>> >>>> 2 >>>> >>>> 2 >>>> >>>> 35383 >>>> >>>> 4.06 >>>> >>>> 0.87 >>>> >>>> 9.71 >>>> >>>> 21.25 >>>> >>>> 70.3 >>>> >>>> 215.9 >>>> >>>> 4 >>>> >>>> 1 >>>> >>>> 34648 >>>> >>>> 2.78 >>>> >>>> 0.86 >>>> >>>> 6.07 >>>> >>>> 14.94 >>>> >>>> 30.8 >>>> >>>> 134.6 >>>> >>>> 4 >>>> >>>> 3 >>>> >>>> 52932 >>>> >>>> 3.45 >>>> >>>> 0.86 >>>> >>>> 10.81 >>>> >>>> 21.05 >>>> >>>> 37.4 >>>> >>>> 189.1 >>>> >>>> >>>> >>>> The test data I use is generated randomly, and the schema I use is lik= e >>>> (I use the cqlsh to create the columnfamily/table): >>>> >>>> CREATE TABLE table( >>>> >>>> id1 varchar, >>>> >>>> ts varchar, >>>> >>>> id2 varchar, >>>> >>>> msg varchar, >>>> >>>> PRIMARY KEY(id1, ts, id2)); >>>> >>>> So the fields are all string and I generate each character of the >>>> string randomly, using srand(time(0)) and rand() in C++, so I think my= test >>>> data could be uniformly distributed into the Cassandra cluster. And, i= n my >>>> client stress test code, I use thrift C++ interface, and the basic >>>> operation I do is like: >>>> >>>> thrift_client.execute_cql3_query(=E2=80=9CINSERT INTO table WHERE id1= =3Dxxx, >>>> ts=3Dxxx, id2=3Dxxx, msg=3Dxxx=E2=80=9D); and thrift_client.execute_cq= l3_query(=E2=80=9CSELECT >>>> FROM table WHERE id1=3Dxxx=E2=80=9D); >>>> >>>> Each data entry I INSERT of SELECT is of around 100 characters. >>>> >>>> On my stress test client, I create several threads to send the read an= d >>>> write requests, each thread having its own thrift client, and at the >>>> beginning all the thrift clients connect to the Cassandra servers even= ly. >>>> For example, I create 160 thrift clients, and each 40 clients of them >>>> connect to one server node, in a 4 node cluster. >>>> >>>> >>>> >>>> *So, * >>>> >>>> *1. **Could anyone help me explain my test results? Why does the >>>> performance ( QPS ) just get a little increment when new nodes are add= ed? * >>>> >>>> *2. **I learn from the materials that, Cassandra has better >>>> write performance than read. But why in my case the read performance i= s >>>> better?* >>>> >>>> *3. **I also use the OpsCenter to monitor the real-time >>>> performance of my cluster. But when I get the average QPS above, the >>>> operations/s provided by OpsCenter is around 10000+ for write peak and >>>> 5000+ for read peak. Why is my result inconsistent with that from >>>> OpsCenter?* >>>> >>>> *4. **Are there any unreasonable things in my test method, such >>>> as test data and QPS calculation?* >>>> >>>> >>>> >>>> *Thank you very much,* >>>> >>>> *Joy* >>>> >>> >> > --001a11c316a260743b0509b24a6b Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thanks Chris.
I run a client on a separate AWS = instance from the Cassandra cluster servers. At the client side, I c= reate 40 or 50 threads for sending requests to each Cassandra node. I creat= e one thrift client for each of the threads. And at the beginning, all the = created thrift clients connect to the corresponding Cassandra nodes and kee= p connecting during the whole process(I did not close all the transports un= til the end of the test process). So I use very simple load balancing, sinc= e the same number of thrift clients connect to each node. And my source cod= e is here: https://github.com/kongjialin/Cassandra/blob/master/ca= ssandra_client.cpp=C2=A0It's very nice of you to help me improve my= code.

As I increase the number of threads, the la= tency gets longer.=C2=A0

I'm using C++, so if = I want to use native binary + prepared statements, the only way is to use C= ++ driver?
Thanks very much.


<= div>

2014-12-08 12:51 GMT+08:00 Chris Lohfink <clohfink85@gmail.com= >:
I think you= r client could use improvements.=C2=A0 How many threads do you have running= in your test?=C2=A0 With a thrift call like that you only can do one reque= st at a time per connection. =C2=A0=C2=A0For example, assuming C* takes 0ms= , a 10ms network latency/driver overhead will mean 20ms RTT and a max throu= ghput of ~50 QPS per thread (native binary doesn't behave like this).= =C2=A0 Are you running client on its own system or shared with a node? =C2= =A0how are you load balancing your requests?=C2=A0 Source code would help s= ince theres a lot that can become a bottleneck.

Generall= y you will see a bit of a dip in latency from N=3DRF=3D1 and N=3D2, RF=3D2 = etc since there are optimizations on the coordinator node when it doesn'= ;t need to send the request to the replicas.=C2=A0 The impact of the networ= k overhead decreases in significance as cluster grows.=C2=A0 Typically; lat= ency wise, RF=3DN=3D1 is going to be fastest possible for smaller loads (ie= when a client cannot fully saturate a single node).=C2=A0

Main thing to expect is that latency will plateau and remain fairl= y constant as load/nodes increase while throughput potential will linearly = (empirically at least) increase.

You should rea= lly attempt it with the native binary + prepared statements, running cql ov= er thrift is far from optimal.=C2=A0 I would recommend using the cassandra-= stress tool if you want to stress test Cassandra (and not your code)=C2=A0<= a href=3D"http://www.datastax.com/dev/blog/improved-cassandra-2-1-stress-to= ol-benchmark-any-schema" target=3D"_blank">http://www.datastax.com/dev/blog= /improved-cassandra-2-1-stress-tool-benchmark-any-schema

=
=3D=3D=3D
Chris Lohfink=C2=A0

On Sun, Dec 7, 2014 at 9:48 PM, =E5=AD=94=E5=98=89=E6=9E=97 <= kongjialin92@gmail.com> wrote:
Hi Eric,
Thank you very much for your reply!
=
Do you mean that I should clear my table after each run? Indeed, I can= see several times of compaction during my test, but could only a few times= compaction affect the performance that much? Also, I can see=C2=A0from the= OpsCenter=C2=A0some ParNew GC happen but no CMS GC happen.

<= /div>
I run my test on EC2 cluster, I think the network could be of hig= h speed with in it. Each Cassandra server has 4 units CPU, 15 GiB memory an= d 80 SSD storage, which is of m3.xlarge type.

As f= or latency, which latency should I care about most? p(99) or p(999)? I want= to get the max QPS under a certain limited latency.

I know my testing scenario are not the common case in production, I just= want to know how much burden my cluster can bear under stress.
<= br>
So, how did you test your cluster that can get 86k writes/sec= ? How many requests did you send to your cluster? Was it also 1 million? Di= d you also use OpsCenter to monitor the real time performance? I also wonde= r why the write and read QPS OpsCenter provide are much lower than what I c= alculate. Could you please describe in detail about your test deployment?= =C2=A0

Thank you very much,
Joy

2014-12-07 23:55 GMT+08:00 Eric Stevens <mightye@gmail.com>= ;:
Hi Joy,

Are you resettin= g your data after each test run?=C2=A0 I wonder if your tests are actually = causing you to fall behind on data grooming tasks such as compaction, and s= o performance suffers for your later tests.

There are so many=C2=A0factors which can affect performance, without reviewing t= est methodology in great detail, it's really hard to say whether there = are flaws which might uncover an antipattern cause atypical number of cache= hits or misses, and so forth. You may also be producing gc pressure in the= write path, and so forth.

I can=C2=A0say t= hat 28k writes per second looks just a little low, but it depends a lot on = your network, hardware, and write patterns (eg, data size).=C2=A0 For a lit= tle performance test suite I wrote, with parallel batched writes, on a 3 no= de rf=3D3 cluster test cluster, I got about 86k writes per second.

Also focusing exclusively on max latency is going to cause= you some troubles especially in the case of magnetic media as you're u= sing.=C2=A0 Between ill-timed GC and inconsistent performance characteristi= cs from magnetic media, your max numbers will often look significantly wors= e than your p(99) or p(999) numbers.

All this said= , one node will often look better than several nodes for certain patterns b= ecause it completely eliminates proxy (coordinator) write times.=C2=A0 All = writes are local writes.=C2=A0 It's an over-simple case that doesn'= t reflect any practical production use of Cassandra, so it's probably n= ot worth even including in your tests.=C2=A0 I would recommend start at 3 n= odes rf=3D3, and compare against 6 nodes rf=3D6.=C2=A0 Make sure you're= staying on top of compaction and aren't seeing garbage collections in = the logs (either of those will be polluting your results with variability y= ou can't account for with small sample sizes of ~1 million).
=
If you expect to sustain write volumes like this, you'll= find these clusters are sized too small (on that hardware you won't ke= ep up with compaction), and your tests are again testing scenarios you woul= dn't actually see in production.

On Sat Dec 06 2014 at 7:09:18 AM kong <kongjialin92@gmail.com> wrote:

Hi,=

I am doing stres= s test on Datastax Cassandra Community 2.1.2, not using the provided stress= test tool, but use my own stress-test client code instead(I write some C++= stress test code). My Cassandra cluster is deployed on Amazon EC2, using t= he provided Datastax Community AMI( HVM instances ) in the Datastax documen= t, and I am not using EBS, just using the ephemeral storage by default. The= EC2 type of Cassandra servers are m3.xlarge. I use another EC2 instance fo= r my stress test client, which is of type r3.8xlarge. Both the Cassandra se= ver nodes and stress test client node are in us-east. I test the Cassandra = cluster which is made up of 1 node, 2 nodes, and 4 nodes separately. I just= do INSERT test and SELECT test separately, but the performance doesn=E2=80= =99t get linear increment when new nodes are added. Also I get some weird r= esults. My test results are as follows(I do 1 million operations and I t= ry to get the best QPS when the max latency is no more than 200ms, and the = latencies are measured from the client side. The QPS is calculated by total= _operations/total_time).

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0

INSERT(write):

<= table border=3D"0" cellspacing=3D"0" cellpadding=3D"0" width=3D"822" style= =3D"width:616.4pt;border-collapse:collapse">

Node count

Replication f= actor

=C2=A0 QPS

Average latency(ms)=

Min latency= (ms)

.95 latency(ms)

.99 latency(ms)

.999 laten= cy(ms)

Max latency(ms)

1=

1

18687<= u>

2.08=

1.48=

2.95=

5.74=

52.8=

=

205.4=

2

<= td width=3D"104" nowrap style=3D"width:77.95pt;border-top:none;border-left:= none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0= pt;padding:0cm 5.4pt 0cm 5.4pt;height:13.5pt">

1

20793

3.15

0.84

7.71

41.35

88.7

232.7

2

2

22498

3.37

0.86

6.04

36.1

221.5

649.3

4

1

28348

4.38

0.85

8.19

64.51<= u>

169.4=

251.9=

4

3

28631

5.22

0.87

18.68

68.35

167.2

288

=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0

SELECT(read):

<= td width=3D"72" nowrap style=3D"width:54.0pt;border:solid windowtext 1.0pt;= border-top:none;padding:0cm 5.4pt 0cm 5.4pt;height:13.5pt">

1

<= td width=3D"102" nowrap style=3D"width:76.8pt;border-top:none;border-left:n= one;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0p= t;padding:0cm 5.4pt 0cm 5.4pt;height:13.5pt">

1

34648

<= td width=3D"104" nowrap style=3D"width:77.95pt;border-top:none;border-left:= none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0= pt;padding:0cm 5.4pt 0cm 5.4pt;height:13.5pt">

2.78

Node count

Replication factor<= /u>

Q= PS

Average latency(ms)

Min latency(ms)

=

.95 la= tency(ms)

.99 latency(ms)

.999 latency(ms)

Max = latency(ms)

1=

24498

4.01

1.51

7.6

12.51

31.5

129.6

2

1

28219

3.38

0.85

9.5

17.71

39.2

152.2

=

2<= /u>

2

35383<= /u>

4.06

=

0.87<= u>

=

9.71<= u>

=

21.25=

70.3=

=

215.9=

4

0.86

6.07

14.94

30.8

134.6

4

3

52932

3.45

0.86

10.81

21.05

37.4

189.1

=C2=A0=

The test data I use is gene= rated randomly, and the schema I use is like (I use the cqlsh to create the= columnfamily/table):

CREATE TABLE table(

id1=C2=A0 varchar,=

ts =C2=A0=C2=A0varchar,

id2=C2= =A0 varchar,

msg =C2=A0varchar,<= /p>

PRIMARY KEY(id1, ts, id2));

So the fields are all string and I generate each char= acter of the string randomly, using srand(time(0)) and rand() in C++, so I = think my test data could be uniformly distributed into the Cassandra cluste= r. And, in my client stress test code, I use thrift C++ interface, and the = basic operation I do is like:

thrift_client.execute_cql3_query(=E2=80=9CINSERT INT= O table WHERE id1=3Dxxx, ts=3Dxxx, id2=3Dxxx, msg=3Dxxx=E2=80=9D); and thri= ft_client.execute_cql3_query(=E2=80=9CSELECT FROM table WHERE id1=3Dxxx=E2= =80=9D);

Each data entry I INSERT of SELECT is of around 100 characters.

On my stress te= st client, I create several threads to send the read and write requests, ea= ch thread having its own thrift client, and at the beginning all the thrift= clients connect to the Cassandra servers evenly. For example, I create 160= thrift clients, and each 40 clients of them connect to one server node, in= a 4 node cluster.

=C2=A0

So,

1.=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Could anyone help me explain my t= est results? Why does the performance ( QPS ) just get a little increment w= hen new nodes are added?

2.=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 I learn from the materials that, Cassandra has better write performance = than read. But why in my case the read performance is better?

3.=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 I also use the OpsCenter to monitor t= he real-time performance of my cluster. But when I get the average QPS abov= e, the operations/s provided by OpsCenter is around 10000+ for write peak a= nd 5000+ for read peak.=C2=A0 Why is my result inconsistent with that from = OpsCenter?

<= /u>4.=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Are there any unreasonable things in my te= st method, such as test data and QPS calculation?<= /p>

=C2=A0

Thank you very muc= h,

Joy




--001a11c316a260743b0509b24a6b--