Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 414E4186DB for ; Tue, 8 Dec 2015 17:15:22 +0000 (UTC) Received: (qmail 124 invoked by uid 500); 8 Dec 2015 17:15:19 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 99981 invoked by uid 500); 8 Dec 2015 17:15:19 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 99971 invoked by uid 99); 8 Dec 2015 17:15:19 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Dec 2015 17:15:19 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id DD8E3C963B for ; Tue, 8 Dec 2015 17:15:18 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.161 X-Spam-Level: *** X-Spam-Status: No, score=3.161 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=3, T_FILL_THIS_FORM_SHORT=0.01, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=yahoo.co.in Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id EO_C1rPJXDED for ; Tue, 8 Dec 2015 17:15:05 +0000 (UTC) Received: from nm14.bullet.mail.sg3.yahoo.com (nm14.bullet.mail.sg3.yahoo.com [106.10.148.93]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id 97DA342B0F for ; Tue, 8 Dec 2015 17:15:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.co.in; s=s2048; t=1449594547; bh=OA+LDT1xINAGoi08gsQIZ3jiPNOtj7G7IWif09uoxD4=; h=Date:From:Subject:To:In-Reply-To:From:Subject; b=FsjWpmqtf6vcJPR0Zw1C/EYM1tJl3242l9auDj0KmljKCpdIKpCEDHTMJr8mwnaSslUUqU6EA+OXfQq5ZXL6+c9mhJtJR0/gu/erg3DH73Swc/xl6Zknu+Umj2yOmUUOy4YvFa0+cGw5B7YKX1iuS4yolWScx3OIpjI4tBxajS8Fgr25pdBZpFaXr4fzlpxb9cx80HK8ZXG8kDQEnoR9fEuYKS6WGKQ+LjxPtvMJXaoKakkoPvOdU9l3ai2MDauNkdGqauirm0Df/FLj2uojXe8OrNpCV1jig5PswHO/BvKoYGcPsIEWVzv9CfJgp+2opeik7gxh0bKU4rI3SDOzeQ== Received: from [106.10.166.62] by nm14.bullet.mail.sg3.yahoo.com with NNFMP; 08 Dec 2015 17:09:07 -0000 Received: from [106.10.151.139] by tm19.bullet.mail.sg3.yahoo.com with NNFMP; 08 Dec 2015 17:09:07 -0000 Received: from [127.0.0.1] by omp1007.mail.sg3.yahoo.com with NNFMP; 08 Dec 2015 17:09:07 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 239445.66074.bm@omp1007.mail.sg3.yahoo.com Received: (qmail 77706 invoked by uid 60001); 8 Dec 2015 17:09:07 -0000 X-YMail-OSG: jz5yiPUVM1m1IK.x2N.nGPQ2duPGhvaWWrL9iJ6G6W6lYUK duIr41aIpubNMSbCHzt9tV5GdTondHv8O7umtkBYxGEnSkqfiN5.JCwbLUWi CK5tPHFtdoolU3SoOfK4zBQtfDBpl1lIAuyXwEixizLD0GbKuYaJgShWRYMt QfKDGz5WEY3iW.50DrNstxKNyLGNhbZq.rZbfYYQxcp9oaxyMKTQQNCbJODh FthI.RicTSdRtYZHEaSthBGpahvfOv8OH8CUHNUK9TSmNna3oWk0p9fEUmyq r.F1hha_0BJw0E0VrqFzeWgSsA97It4t1LyAtglOVsWoyuidt4DTK2U6unzy zjdP1F40ytqp3sB_x6SGvtTK9_SUqDE4avek8C_muyl3PP1zK0kVzX.abroB 7l3nFVYYs15zMS_.aJmqpwmfbiP.VyKt5Bdnvl902cJaBUlj8l.uuR2STkNY atFaM2kQH4UYtEqKcrFJ94sGeGer.9yudCjaP8BwWJH4BijSh4VjkhHnd5_Y 69.RtGvKUwQhDGN3GeUrnPbphKs9.dSp2m1qryMEwRTtI8yytNrFG.gEMSkJ 4mBzcAGbze3IrJJlb2jpDS.SxDL8- Received: from [150.107.8.242] by web192903.mail.sg3.yahoo.com via HTTP; Wed, 09 Dec 2015 01:09:07 SGT X-Rocket-MIMEInfo: 002.001,SGkgSmVycnksCgoKSXRzIGdyZWF0IHRoYXQgeW91IGdvdCBwZXJmb3JtYW5jZSBpbXByb3ZlbWVudC4gTW9yZW92ZXIsIEkgYWdyZWUgd2l0aCB3aGF0IEdyYWhhbSBzYWlkLiBJIHRoaW5rIHRoYXQgeW91IGFyZSB1c2luZyBleHRyZW1lbHkgbGFyZ2UgSGVhcHMgd2l0aCBDTVMgYW5kIHRoYXQgdG9vIGluIHZlcnkgb2RkIHJhdGlvLi5IYXZpbmcgNDBHIGZvciBuZXcgZ2VuIGFuZCBsZWF2aW5nIG9ubHkgMjBHIG9sZCBnZW4gc2VlbXMgdW5yZWFzb25hYmxlLi5JdHMgaGFyZCB0byBiZWxpZXZlIHRoYXQgeW8BMAEBAQE- X-Mailer: YahooMailAndroidMobile/4.9.2 YahooMailWebService/0.8.203.822 Message-ID: <1449594547.24150.YahooMailAndroidMobile@web192903.mail.sg3.yahoo.com> Date: Wed, 9 Dec 2015 01:09:07 +0800 From: Anuj Wadehra Subject: Re: Re: Re: Cassandra Tuning Issue To: "user@cassandra.apache.org" , "user@cassandra.apache.org" In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="-1985756734-1664357670-1449594547=:24150" ---1985756734-1664357670-1449594547=:24150 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi Jerry,=0A=0A=0AIts great that you got performance improvement. Moreover,= I agree with what Graham said. I think that you are using extremely large = Heaps with CMS and that too in very odd ratio..Having 40G for new gen and l= eaving only 20G old gen seems unreasonable..Its hard to believe that you ar= e having reasonable Gc pauses..Please recheck..I would suggest you to test = your performance with much smaller heap..may be 16G max heap n 4G new gen..= moreover make sure that you apply all the recommended Production settings s= uggested by DataStax at http://docs.datastax.com/en/cassandra/2.1/cassandra= /install/installRecommendSettings.html=0A=0A=0ADont worry about wasting you= r memory, it will be used for OS caching and you can get even better perfor= mance..=0A=0A=0AThanks=0A=0AAnuj=0A=0ASent from Yahoo Mail on Android=0A=0A= From:"Jack Krupansky" =0ADate:Tue, 8 Dec, 2015 at= 8:07 pm=0ASubject:Re: Re: Re: Cassandra Tuning Issue=0A=0AGreat! Make sure= to inform the C* email list as well so that others know.=0A=0A=0A-- Jack K= rupansky=0A=0A=0AOn Tue, Dec 8, 2015 at 7:44 AM, xutom = wrote:=0A=0A=0A=0ADear Jack,=0A=C2=A0=C2=A0=C2=A0 Thank you very much! Now = we have much better performance when we insert the same partition keys in t= he same batch.=0A=0Ajerry=0A=0A=0AAt 2015-12-07 13:08:31, "Jack Krupansky" = wrote:=0A=0AIf you combine inserts for multiple = partition keys in the same batch you negate most of the effect of token-awa= re routing. It's best to insert only rows with the same partition key in a = single batch. You also need to set the partition key for routing for the ba= tch.=0A=0A=0AAlso, RF=3D2 is not recommended since it does not permit quoru= m operations if a replica node is down. RF=3D3 is generally more appropriat= e.=0A=0A=0A-- Jack Krupansky=0A=0A=0AOn Sun, Dec 6, 2015 at 10:27 PM, xutom= wrote:=0A=0ADear all,=0A=C2=A0=C2=A0=C2=A0 Thanks for = ur reply!=0A=C2=A0=C2=A0=C2=A0 Now I`m using Apache Cassandra 2.1.1 and my = JDK is 1.7.0_79,=C2=A0 my keyspace replication factor is 2=EF=BC=8Cand I do= enable the "token aware". The GC configuration is default for such as:=0A#= GC tuning options=0AJVM_OPTS=3D"$JVM_OPTS -XX:+UseParNewGC"=0AJVM_OPTS=3D"= $JVM_OPTS -XX:+UseConcMarkSweepGC"=0AJVM_OPTS=3D"$JVM_OPTS -XX:+CMSParallel= RemarkEnabled"=0A=C2=A0=C2=A0=C2=A0 And I check the gc log: gc.log.0.curren= t, I found there is only one Full GC. The stop-the-world times is low.=0ACM= S-initial-mark: 0.2747280 secs=0ACMS-remark: 0.3623090 secs=0A=0A=C2=A0=C2= =A0=C2=A0 The insert codes in my test client are following:=0A=C2=A0=C2=A0= =C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 String content =3D RandomStrin= gUtils.randomAlphabetic(120);=0A=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2= =A0=C2=A0=C2=A0 cluster =3D Cluster=0A=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 .builder()=0A=C2= =A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 = =C2=A0=C2=A0=C2=A0 .addContactPoint(this.seedIP)=0A=C2=A0=C2=A0=C2=A0 =C2= =A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 .w= ithCredentials("test", "test")=0A=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2= =A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 .withRetryPolicy(Defa= ultRetryPolicy.INSTANCE)=0A=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2= =A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 .withLoadBalancingPolicy(ne= w TokenAwarePolicy(new DCAwareRoundRobinPolicy())) =0A=C2=A0=C2=A0=C2=A0 = =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0= .build();=0A=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 sessi= on =3D cluster.connect("demo");=0A=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2= =A0=C2=A0=C2=A0 ......=0A=0A=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2= =A0=C2=A0 PreparedStatement insertPreparedStatement =3D session.prepare(=0A= =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 "=C2=A0=C2=A0 INSERT INTO teacher (i= d, lastname, firstname, city) " +=0A=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 = =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "VALUES (?,= ?, ?, ?); ");=0A=0A=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 BatchStatement batch =3D new BatchStatement();=0A=C2=A0=C2=A0=C2=A0 = =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 for (; i < max; i+=3D5) {=0A=C2=A0=C2= =A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 try {=0A= =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 batch.add(insertPreparedStatement.bind(i, "Entre Nous",= "adsfasdfa1", content));=0A=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2= =A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 batch.add(insertPreparedSta= tement.bind(i+1, "Entre Nous", "adsfasdfa2", content));=0A=C2=A0=C2=A0=C2= =A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0= =C2=A0 batch.add(insertPreparedStatement.bind(i+2, "Entre Nous", "adsfasdfa= 3", content));=0A=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 = =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 batch.add(insertPreparedStatement.bin= d(i+3, "Entre Nous", "adsfasdfa4", content));=0A=C2=A0=C2=A0=C2=A0 =C2=A0= =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 batch= .add(insertPreparedStatement.bind(i+4, "Entre Nous", "adsfasdfa5", content)= );=0A=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0= =C2=A0 =C2=A0=C2=A0=C2=A0 =0A//=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0= =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 System.out.println("the = is is " + i);=0A=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 = =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 session.execute(batch);=0A=0A=C2=A0= =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2= =A0=C2=A0=C2=A0 thisTimeCount +=3D 5;=0A=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2= =A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 }=0A=C2=A0=C2=A0=C2=A0 =C2=A0=C2= =A0=C2=A0 =C2=A0=C2=A0=C2=A0 }=0A=0A=0A=0A=0AAt 2015-12-07 00:40:06, "Graha= m Sanderson" wrote:=0A=0AWhat version of C* are you using= ; what JVM version - you showed a partial GC config but if that is still CM= S (not G1) then you are going to have insane GC pauses...=C2=A0=0A=0A=0ADep= ending on C* versions are you using on/off heap memtables and what type=0A= =0A=0AThose are the sorts of issues related to fat nodes; I'd be worried ab= out - we run very nicely at 20G total heap and 8G new - the rest of our 128= G memory is disk cache/mmap and all of the off heap stuff so it doesn't go = to waste=0A=0A=0AThat said I think Jack is probably on the right path with = overloaded coordinators- though you'd still expect to see CPU usage unless = your timeouts are too low for the load, In which case the coordinator would= be getting no responses in time and quite possibly the other nodes are jus= t dropping the mutations (since they don't get to them before they know the= coordinator would have timed out) - I forget the command to check dropped = mutations off the top of my head but you can see it in opcenter=0A=0A=0AIf = you have GC problems you certainly=0A=0AExpect to see GC cpu usage but depe= nding on how long you run your tests it might take you a little while to ru= n thru 40G=0A=0A=0AI'm personally not a fan off >32G (ish) heaps as you can= 't do compressed oops and also it is unrealistic for CMS ... The word is th= at G1 is now working ok with C* especially on newer C* and JDK versions, bu= t that said it takes quite a lot of thru-put to require insane quantities o= f young gen... We are guessing that when we remove all our legacy thrift ba= tch inserts we will need less - and as for 20G total we actually don't need= that much (we dropped from 24 when we moved memtables off heap, and believ= e we can drop further)=0A=0A=0ASent from my iPhone=0A=0A=0AOn Dec 6, 2015, = at 9:07 AM, Jack Krupansky wrote:=0A=0AWhat repl= ication factor are you using? Even if your writes use CL.ONE, Cassandra wil= l be attempting writes to the replica nodes in the background.=0A=0A=0AAre = your writes "token aware"? If not, the receiving node has the overhead of f= orwarding the request to the node that owns the token for the primary key.= =0A=0A=0AFor the record, Cassandra is not designed and optimized for so-cal= led "fat nodes". The design focus is "commodity hardware" and "distributed = cluster" (typically a dozen or more nodes.)=0A=0A=0AThat said, it would be = good if we had a rule of thumb for how many simultaneous requests a node ca= n handle, both external requests and inter-node traffic. I think there is a= n open Jira to enforce a limit on inflight requests so that nodes don't ove= rloaded and start failing in the middle of writes as you seem to be seeing.= =0A=0A=0A-- Jack Krupansky=0A=0A=0AOn Sun, Dec 6, 2015 at 9:29 AM, jerry wrote:=0A=0ADear All,=0A=0A=C2=A0 =C2=A0 Now I have a 4 n= odes Cassandra cluster, and I want to know the highest performance of my Ca= ssandra cluster. I write a JAVA client to batch insert datas into ALL 4 nod= es Cassandra, when I start less than 30 subthreads in my client application= s to insert datas into cassandra, it will be ok for everything, but when I = start more than 80 or 100 subthreads in my client applications, there will = be too much timeout Exceptions (Such as: Cassandra timeout during write que= ry at consistency ONE (1 replica were required but only 0 acknowledged the = write)). And no matter how many subthreads or even I start multiple clients= with multiple subthreads on different computers, I can get the highest per= formance for about 60000 - 80000 TPS. By the way, each row I insert into ca= ssandra is about 130 Bytes.=0A=C2=A0 =C2=A0 My 4 nodes of Cassandra is :=0A= =C2=A0 =C2=A0 =C2=A0 =C2=A0 CPU: 4*15=0A=C2=A0 =C2=A0 =C2=A0 =C2=A0 Memory:= 512G=0A=C2=A0 =C2=A0 =C2=A0 =C2=A0 Disk: flash card (only one disk but bet= ter than SSD)=0A=C2=A0 =C2=A0 My cassandra configurations are:=0A=C2=A0 =C2= =A0 =C2=A0 =C2=A0 MAX_HEAP_SIZE: 60G=0A=C2=A0 =C2=A0 =C2=A0 =C2=A0 NEW_HEAP= _SIZE: 40G=0A=0A=C2=A0 =C2=A0 When I insert datas into my cassandra cluster= , each nodes has NOT reached bottleneck such as CPU or Memory or Disk. Each= of the three main hardwares is idle=E3=80=82So I think maybe there is some= thing wrong about my configuration of cassandra cluster. Can somebody pleas= e help me to My Cassandra Tuning? Thanks in advances!=0A=0A=0A=0A=0A=C2=A0= =0A=0A=0A=0A=0A=C2=A0=0A=0A=0A ---1985756734-1664357670-1449594547=:24150 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable
Hi Jerry,
Its great= that you got performance improvement. Moreover, I agree with what Graham s= aid. I think that you are using extremely large Heaps with CMS and that too= in very odd ratio..Having 40G for new gen and leaving only 20G old gen see= ms unreasonable..Its hard to believe that you are having reasonable Gc paus= es..Please recheck..I would suggest you to test your performance with much = smaller heap..may be 16G max heap n 4G new gen..moreover make sure that you= apply all the recommended Production settings suggested by DataStax at htt= p://docs.datastax.com/en/cassandra/2.1/cassandra/install/installRecommendSe= ttings.html

Dont = worry about wasting your memory, it will be used for OS caching and you can get = even better performance..

Thanks
Anuj
<= p>Sent f= rom Yahoo Mail on Android

From:"Jack Krupansky" <ja= ck.krupansky@gmail.com>
Date:Tue, 8 Dec, 2015 at 8:07 pm
Subject:Re: Re: Re: Cassandra Tuning Issue

Great! Make sure to inform the C* email list as well so that others kno= w.

-- Jack Krupansky
=0A
On Tue, Dec 8, 2015 at 7:44 AM, xutom <xutom2006@126.com> wrote:


Dear Jack,
    Thank yo= u very much! Now we have much better performance when we insert the same pa= rtition keys in the same batch.

jerry
At 2015-12-07 13:08:31, "Jack Krupansky" <jack.krupansky@gmail.com> wrote:
=
If you combine inserts for multiple part= ition keys in the same batch you negate most of the effect of token-aware r= outing. It's best to insert only rows with the same partition key in a sing= le batch. You also need to set the partition key for routing for the batch.=

Also, RF=3D2 is not recommended since it= does not permit quorum operations if a replica node is down. RF=3D3 is gen= erally more appropriate.

-- Jack Krupansky
=0A
On Sun, Dec 6, 2015 at 10:27 PM, = xutom <xutom= 2006@126.com> wrote:
Dear all,
    Thanks fo= r ur reply!
    Now I`m using Apache Cassa= ndra 2.1.1 and my JDK is 1.7.0_79,  my keyspace replication factor is = 2=EF=BC=8Cand I do enable the "token aware". The GC configuration is defaul= t for such as:
# GC tuning options
JVM_= OPTS=3D"$JVM_OPTS -XX:+UseParNewGC"
JVM_OPTS=3D"$JVM_OPTS= -XX:+UseConcMarkSweepGC"
JVM_OPTS=3D"$JVM_OPTS -XX:+CMSP= arallelRemarkEnabled"
    And I check the gc log: gc.log.0.current= , I found there is only one Full GC. The stop-the-world times is low.
CMS-initial-mark: 0.2747280 secs
CMS-remark:= 0.3623090 secs

    The= insert codes in my test client are following:
 &nbs= p;          String content =3D RandomStr= ingUtils.randomAlphabetic(120);
     =        cluster =3D Cluster
 = ;               &nbs= p;   .builder()
      =               .addContact= Point(this.seedIP)
        =             .withCredentials("= test", "test")
            &n= bsp;       .withRetryPolicy(DefaultRetryPolicy.INS= TANCE)
          =           .withLoadBalancingPolicy(new T= okenAwarePolicy(new DCAwareRoundRobinPolicy()))
 &n= bsp;               &= nbsp;  .build();
      &nbs= p;     session =3D cluster.connect("demo");
            ......
            Pr= eparedStatement insertPreparedStatement =3D session.prepare(
              &n= bsp;         "   INSERT INTO teache= r (id, lastname, firstname, city) " +
     =              &n= bsp;            = ; "VALUES (?, ?, ?, ?); ");

 &nbs= p;          BatchStatement batch = =3D new BatchStatement();
      =       for (; i < max; i+=3D5) {
&n= bsp;               t= ry {
          &n= bsp;         batch.add(insertPreparedStatemen= t.bind(i, "Entre Nous", "adsfasdfa1", content));
 &n= bsp;               &= nbsp;  batch.add(insertPreparedStatement.bind(i+1, "Entre Nous", "adsf= asdfa2", content));
         &= nbsp;          batch.add(insertPreparedS= tatement.bind(i+2, "Entre Nous", "adsfasdfa3", content));
               = ;     batch.add(insertPreparedStatement.bind(i+3, "Entre Nou= s", "adsfasdfa4", content));
     &nb= sp;              batch.ad= d(insertPreparedStatement.bind(i+4, "Entre Nous", "adsfasdfa5", content));<= br clear=3D"none">            =        
//  &nbs= p;               &nb= sp; System.out.println("the is is " + i);
  &nb= sp;               &n= bsp; session.execute(batch);
     &n= bsp;              thisTim= eCount +=3D 5;
        &nbs= p;       }
    &n= bsp;       }

=
At 2015-12-07 00:40:06, "Graham Sanderson" <graham@vast.com> wrote:
What version of C* are you using; what JVM vers= ion - you showed a partial GC config but if that is still CMS (not G1) then= you are going to have insane GC pauses... 

Depending on C* versions are you using on/off heap memtables and what type
Those are the sorts of issues related to fat no= des; I'd be worried about - we run very nicely at 20G total heap and 8G new= - the rest of our 128G memory is disk cache/mmap and all of the off heap s= tuff so it doesn't go to waste

That= said I think Jack is probably on the right path with overloaded coordinato= rs- though you'd still expect to see CPU usage unless your timeouts are too= low for the load, In which case the coordinator would be getting no respon= ses in time and quite possibly the other nodes are just dropping the mutati= ons (since they don't get to them before they know the coordinator would ha= ve timed out) - I forget the command to check dropped mutations off the top= of my head but you can see it in opcenter

If you have GC problems you certainly
Expect to see GC cpu usage but depending on how long you run your tests it might take you a= little while to run thru 40G

I'm p= ersonally not a fan off >32G (ish) heaps as you can't do compressed oops= and also it is unrealistic for CMS ... The word is that G1 is now working = ok with C* especially on newer C* and JDK versions, but that said it takes = quite a lot of thru-put to require insane quantities of young gen... We are= guessing that when we remove all our legacy thrift batch inserts we will n= eed less - and as for 20G total we actually don't need that much (we droppe= d from 24 when we moved memtables off heap, and believe we can drop further= )

Sent from my iPhone

On Dec 6, 2015, at 9:07 AM, Jack Krupansky <jack.krupansky@gmail.com> wrote:

What replication factor are you using? Even if your writes us= e CL.ONE, Cassandra will be attempting writes to the replica nodes in the b= ackground.

Are your writes "token aware"?= If not, the receiving node has the overhead of forwarding the request to t= he node that owns the token for the primary key.

For the record, Cassandra is not designed and optimized for s= o-called "fat nodes". The design focus is "commodity hardware" and "distrib= uted cluster" (typically a dozen or more nodes.)

That said, it would be good if we had a rule of thumb for how= many simultaneous requests a node can handle, both external requests and i= nter-node traffic. I think there is an open Jira to enforce a limit on infl= ight requests so that nodes don't overloaded and start failing in the middl= e of writes as you seem to be seeing.
-- Jack Krupansky
=0A
On Sun, Dec 6, 2015 at 9= :29 AM, jerry <xutom2006@126.com> wrote:
Dea= r All,
=0A
=0A    Now I have = a 4 nodes Cassandra cluster, and I want to know the highest performance of = my Cassandra cluster. I write a JAVA client to batch insert datas into ALL = 4 nodes Cassandra, when I start less than 30 subthreads in my client applic= ations to insert datas into cassandra, it will be ok for everything, but wh= en I start more than 80 or 100 subthreads in my client applications, there = will be too much timeout Exceptions (Such as: Cassandra timeout during writ= e query at consistency ONE (1 replica were required but only 0 acknowledged= the write)). And no matter how many subthreads or even I start multiple cl= ients with multiple subthreads on different computers, I can get the highes= t performance for about 60000 - 80000 TPS. By the way, each row I insert in= to cassandra is about 130 Bytes.
=0A    My 4 no= des of Cassandra is :
=0A        CPU:= 4*15
=0A        Memory: 512G
=0A        Disk: flash card (only one disk b= ut better than SSD)
=0A    My cassandra configu= rations are:
=0A        MAX_HEAP_SIZE= : 60G
=0A        NEW_HEAP_SIZE: 40G=0A
=0A    When I insert datas= into my cassandra cluster, each nodes has NOT reached bottleneck such as C= PU or Memory or Disk. Each of the three main hardwares is idle=E3=80=82So I= think maybe there is something wrong about my configuration of cassandra c= luster. Can somebody please help me to My Cassandra Tuning? Thanks in advan= ces!
=0A

=0A


 

<= /div>
=0A


&n= bsp;
---1985756734-1664357670-1449594547=:24150--