Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5ED271071A for ; Thu, 31 Oct 2013 18:12:47 +0000 (UTC) Received: (qmail 67980 invoked by uid 500); 31 Oct 2013 18:12:44 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 67901 invoked by uid 500); 31 Oct 2013 18:12:44 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 67893 invoked by uid 99); 31 Oct 2013 18:12:44 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Oct 2013 18:12:44 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of tyagi.iitr@gmail.com designates 209.85.128.181 as permitted sender) Received: from [209.85.128.181] (HELO mail-ve0-f181.google.com) (209.85.128.181) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Oct 2013 18:12:38 +0000 Received: by mail-ve0-f181.google.com with SMTP id jz11so2391391veb.12 for ; Thu, 31 Oct 2013 11:12:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=yPomLfGy5vx04DynFib7csx0rDUOPrnDhicIOMDpLiA=; b=rk+EIvcpMcXhpPdLO/5rp9I1NOg7brqzha4aBkRbawNGimIAkfN0+bT1eyXqJDMFTk W/BZCqfEwguu30MfBHAHn6m9egBuoKpRP3+IINgjNGxuem/AJhN09MjU/T2T+ZrzTeNI C3l9Surl9QuIHPp5hBenCjVcUPOEWhCTs569IKxsDsFu10QVIDqxNunbHXIkt1xmGcOz I10bmzBFHbJZpOhPz+E828Vp266MTCvu06Zi5QEl1B6daBhlxvFaNH8yYOhXqbm9tev1 EBFu62tgN7IA3WCx7eXAhe0gCeYcoPEx9PLboyex6N/7MVnImlm0Ddwy8XRXH+d4adBK 7fPg== MIME-Version: 1.0 X-Received: by 10.58.117.7 with SMTP id ka7mr334297veb.44.1383243137423; Thu, 31 Oct 2013 11:12:17 -0700 (PDT) Received: by 10.59.10.99 with HTTP; Thu, 31 Oct 2013 11:12:17 -0700 (PDT) Date: Thu, 31 Oct 2013 23:42:17 +0530 Message-ID: Subject: High loads only on one node in the cluster From: Ashish Tyagi To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=047d7b86f130d821e204ea0d6348 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b86f130d821e204ea0d6348 Content-Type: text/plain; charset=ISO-8859-1 We have a 9 node cluster. 6 nodes are in one data-center and 3 nodes in the other. All machines are Amazon M1.XLarge configuration. Datacenter: DC1 ========== Address Rack Status State Load Owns Token ip11 1b Up Normal 76.46 GB 16.67% 0 ip12 1b Up Normal 44.66 GB 16.67% 28356863910078205288614550619314017621 ip13 1c Up Normal 85.94 GB 16.67% 56713727820156410577229101238628035241 ip14 1c Up Normal 17.55 GB 16.67% 85070591730234615865843651857942052863 ip15 1d Up Normal 80.74 GB 16.67% 113427455640312821154458202477256070484 ip16 1d Up Normal 20.88 GB 16.67% 141784319550391026443072753096570088105 Datacenter: DC2 ========== Address Rack Status State Load Owns Token ip21 1a Up Normal 78.32 GB 0.00% 1001 ip22 1b Up Normal 71.23 GB 0.00% 56713727820156410577229101238628036241 ip23 1b Up Normal 53.49 GB 0.00% 113427455640312821154458202477256071484 Problem is that node with ip address: ip11 often has 5-10 times more load than any other node. Most of the operations are on counters. The primary column family (which receives most writes) has a replication factor of 2 in DataCenter DC1 and also in DataCenter DC2. The traffic is write heavy (reads are less than 10% of total requests). We are using size-tiered compaction. Both writes and reads happen with a consistency factor of LOCAL_QUORUM. More information: 1. cassandra.yaml - http://pastebin.com/u344fA6z 2. Jmap heap when node under high loads - http://pastebin.com/ib3D0Pa 3. Nodetool tpstats - http://pastebin.com/s0AS7bGd 4. Cassandra-env.sh - http://pastebin.com/ubp4cGUx 5. GC log lines - http://pastebin.com/Y0TKphsm Am I doing anything wrong. Any pointers will be appreciated. Thanks in advance, Ashish --047d7b86f130d821e204ea0d6348 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: base64 PGRpdiBkaXI9Imx0ciI+PGRpdj48ZGl2PjxkaXY+PGRpdj5XZSBoYXZlIGEgOSBub2RlIGNsdXN0 ZXIuIDYgbm9kZXMgYXJlIGluIG9uZSBkYXRhLWNlbnRlciBhbmQgMyBub2RlcyBpbiB0aGUgb3Ro ZXIuIEFsbCBtYWNoaW5lcyBhcmUgQW1hem9uIE0xLlhMYXJnZSBjb25maWd1cmF0aW9uLjxicj48 YnI+RGF0YWNlbnRlcjogREMxPGJyPj09PT09PT09PT08YnI+QWRkcmVzc6CgoKCgoKCgIFJhY2ug oKCgoKCgIFN0YXR1cyBTdGF0ZaCgIExvYWSgoKCgoKCgoKCgoCBPd25zoKCgoKCgoKCgoKCgoKCg IFRva2VuoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKAgPGJyPg0KDQqgoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKCgoCA8YnI+aXAxMaAgMWKgoKCgoKCgoKAgVXCgoKCgIE5vcm1h bKAgNzYuNDYgR0KgoKCgoKCgIDE2LjY3JaCgoKCgoKCgoKCgoKAgMKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoCA8YnI+aXAxMqAgMWKgoKCgoKCgoKAgVXCgoKCgIE5v cm1hbKAgNDQuNjYgR0KgoKCgoKCgIDE2LjY3JaCgoKCgoKCgoKCgoKAgMjgzNTY4NjM5MTAwNzgy MDUyODg2MTQ1NTA2MTkzMTQwMTc2MjGgoKCgoCA8YnI+DQoNCmlwMTOgIDFjoKCgoKCgoKCgIFVw oKCgoCBOb3JtYWygIDg1Ljk0IEdCoKCgoKCgoCAxNi42NyWgoKCgoKCgoKCgoKCgIDU2NzEzNzI3 ODIwMTU2NDEwNTc3MjI5MTAxMjM4NjI4MDM1MjQxoKCgoKAgPGJyPmlwMTSgIDFjoKCgoKCgoKCg IFVwoKCgoCBOb3JtYWygIDE3LjU1IEdCoKCgoKCgoCAxNi42NyWgoKCgoKCgoKCgoKCgIDg1MDcw NTkxNzMwMjM0NjE1ODY1ODQzNjUxODU3OTQyMDUyODYzoKCgoKAgPGJyPg0KDQppcDE1oCAxZKCg oKCgoKCgoCBVcKCgoKAgTm9ybWFsoCA4MC43NCBHQqCgoKCgoKAgMTYuNjcloKCgoKCgoKCgoKCg oCAxMTM0Mjc0NTU2NDAzMTI4MjExNTQ0NTgyMDI0NzcyNTYwNzA0ODSgoKCgIDxicj5pcDE2oCAx ZKCgoKCgoKCgoCBVcKCgoKAgTm9ybWFsoCAyMC44OCBHQqCgoKCgoKAgMTYuNjcloKCgoKCgoKCg oKCgoCAxNDE3ODQzMTk1NTAzOTEwMjY0NDMwNzI3NTMwOTY1NzAwODgxMDWgoKCgIDxicj4NCg0K PGJyPkRhdGFjZW50ZXI6IERDMjxicj49PT09PT09PT09PGJyPkFkZHJlc3OgoKCgoKCgoCBSYWNr oKCgoKCgoCBTdGF0dXMgU3RhdGWgoCBMb2FkoKCgoKCgoKCgoKAgT3duc6CgoKCgoKCgoKCgoKCg oCBUb2tlbqCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgIDxicj6goKCgoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgIDxicj4NCg0KaXAyMaAgMWGgoKCgoKCgoKAgVXCgoKCgIE5vcm1hbKAgNzguMzIg R0KgoKCgoKCgIDAuMDAloKCgoKCgoKCgoKCgoKAgMTAwMaCgoKCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoKCgoCA8YnI+aXAyMqAgMWKgoKCgoKCgoKAgVXCgoKCgIE5vcm1hbKAgNzEu MjMgR0KgoKCgoKCgIDAuMDAloKCgoKCgoKCgoKCgoKAgNTY3MTM3Mjc4MjAxNTY0MTA1NzcyMjkx MDEyMzg2MjgwMzYyNDGgoKCgoCA8YnI+DQoNCmlwMjOgIDFioKCgoKCgoKCgIFVwoKCgoCBOb3Jt YWygIDUzLjQ5IEdCoKCgoKCgoCAwLjAwJaCgoKCgoKCgoKCgoKCgIDExMzQyNzQ1NTY0MDMxMjgy MTE1NDQ1ODIwMjQ3NzI1NjA3MTQ4NCA8YnI+PGJyPjwvZGl2PlByb2JsZW0gaXMgdGhhdCBub2Rl IHdpdGggaXAgYWRkcmVzczogaXAxMSBvZnRlbiBoYXMgNS0xMCB0aW1lcyBtb3JlIGxvYWQgdGhh biBhbnkgb3RoZXIgbm9kZS4gTW9zdCBvZiB0aGUgb3BlcmF0aW9ucyBhcmUgb24gY291bnRlcnMu IFRoZSBwcmltYXJ5IGNvbHVtbiBmYW1pbHkgKHdoaWNoIHJlY2VpdmVzIG1vc3Qgd3JpdGVzKSBo YXMgYSByZXBsaWNhdGlvbiBmYWN0b3Igb2YgMiBpbiBEYXRhQ2VudGVyIERDMSBhbmQgYWxzbyBp biBEYXRhQ2VudGVyIERDMi4gVGhlIHRyYWZmaWMgaXMgd3JpdGUgaGVhdnkgKHJlYWRzIGFyZSBs ZXNzIHRoYW4gMTAlIG9mIHRvdGFsIHJlcXVlc3RzKS4gV2UgYXJlIHVzaW5nIHNpemUtdGllcmVk IGNvbXBhY3Rpb24uIEJvdGggd3JpdGVzIGFuZCByZWFkcyBoYXBwZW4gd2l0aCBhIGNvbnNpc3Rl bmN5IGZhY3RvciBvZiBMT0NBTF9RVU9SVU0uIDxicj4NCg0KPGJyPjwvZGl2PjxkaXY+TW9yZSBp bmZvcm1hdGlvbjo8YnI+PGJyPjEuIGNhc3NhbmRyYS55YW1sIC0gPGEgaHJlZj0iaHR0cDovL3Bh c3RlYmluLmNvbS91MzQ0ZkE2eiIgdGFyZ2V0PSJfYmxhbmsiPmh0dHA6Ly9wYXN0ZWJpbi5jb20v dTM0NGZBNno8L2E+IDxicj48L2Rpdj4yLiBKbWFwIGhlYXAgd2hlbiBub2RlIHVuZGVyIGhpZ2gg bG9hZHMgLSA8YSBocmVmPSJodHRwOi8vcGFzdGViaW4uY29tL2liM0QwUGEiIHRhcmdldD0iX2Js YW5rIj5odHRwOi8vcGFzdGViaW4uY29tL2liM0QwUGE8L2E+PGJyPg0KMy4gTm9kZXRvb2wgdHBz dGF0cyAtIDxhIGhyZWY9Imh0dHA6Ly9wYXN0ZWJpbi5jb20vczBBUzdiR2QiIHRhcmdldD0iX2Js YW5rIj5odHRwOi8vcGFzdGViaW4uY29tL3MwQVM3YkdkPC9hPjxicj40LiBDYXNzYW5kcmEtZW52 LnNoIC0gPGEgaHJlZj0iaHR0cDovL3Bhc3RlYmluLmNvbS91YnA0Y0dVeCIgdGFyZ2V0PSJfYmxh bmsiPmh0dHA6Ly9wYXN0ZWJpbi5jb20vdWJwNGNHVXg8L2E+PGJyPg0KNS4gR0MgbG9nIGxpbmVz IC2gIDxhIGhyZWY9Imh0dHA6Ly9wYXN0ZWJpbi5jb20vWTBUS3Boc20iPmh0dHA6Ly9wYXN0ZWJp bi5jb20vWTBUS3Boc208L2E+IDxicj48YnI+PC9kaXY+QW0gSSBkb2luZyBhbnl0aGluZyB3cm9u Zy4gQW55IHBvaW50ZXJzIHdpbGwgYmUgYXBwcmVjaWF0ZWQuPGJyPg0KPGJyPjwvZGl2PlRoYW5r cyBpbiBhZHZhbmNlLDxicj5Bc2hpc2g8L2Rpdj4NCg== --047d7b86f130d821e204ea0d6348--