Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E4A10E0FC for ; Wed, 20 Feb 2013 23:08:10 +0000 (UTC) Received: (qmail 35532 invoked by uid 500); 20 Feb 2013 23:08:08 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 35507 invoked by uid 500); 20 Feb 2013 23:08:08 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 35496 invoked by uid 99); 20 Feb 2013 23:08:08 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Feb 2013 23:08:08 +0000 X-ASF-Spam-Status: No, hits=-2.3 required=5.0 tests=RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [192.174.58.133] (HELO XEDGEB.nrel.gov) (192.174.58.133) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Feb 2013 23:08:03 +0000 Received: from XHUBA.nrel.gov (10.20.4.58) by XEDGEB.nrel.gov (192.174.58.133) with Microsoft SMTP Server (TLS) id 8.3.245.1; Wed, 20 Feb 2013 16:07:37 -0700 Received: from MAILBOX2.nrel.gov ([fe80::19a0:6c19:6421:12f]) by XHUBA.nrel.gov ([::1]) with mapi; Wed, 20 Feb 2013 16:07:41 -0700 From: "Hiller, Dean" To: "user@cassandra.apache.org" Date: Wed, 20 Feb 2013 16:07:41 -0700 Subject: Re: load on cluster nodes Thread-Topic: load on cluster nodes Thread-Index: Ac4PvxVLO2HrPrQPQF2fBjETnq/O0g== Message-ID: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.2.5.121010 acceptlanguage: en-US Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org Check the logs for compaction. CPU will go up while one node may be compac= ting and the other node is not compacting yet. Compaction can also last qu= ite a long time in some cases. Also, you can do a jstack =96l {pid} > thre= ad.txt to do a thread dump to see what that node is doing and compare it wi= th other nodes. I am actually in the process of doing quite a bit of performance analysis m= yself and I know I need to go back and tweak the astyanax client somehow as= well as I found I can run a test of running jmap heap dump on node 1 in my= cluster and our website sometimes slows down(ie. I keep refreshing the pag= es) and I feel astyanax is going back to node 1 too often which results in = a 10 second page load vs. when it hits nodes that are fine, it loads in 300= ms. We have been running live for a while and get various percentage bumps on s= ome nodes vs. others but mostly has been compaction running at different ti= mes for quite a while. Dean From: Marcelo Elias Del Valle > Reply-To: "user@cassandra.apache.org" > Date: Wednesday, February 20, 2013 3:53 PM To: "user@cassandra.apache.org" > Subject: load on cluster nodes Hello, I am writing a lot of data on a cassandra cluster with 3 nodes. All my= three nodes are seeds, when I create the astyanax connection pool I specif= y ip1,ip2,ip3 as seeds and so in the casandra.yaml config file. However, if I look at the server load when I am bombarding it with da= ta, I can see the first server is clearly being more demanded (graph here: = http://mvalle.com/downloads/serverload1.png) . I am writting data in sequen= ce, generating UUIDs for column families based on timestamp. Also, I am usi= ng default partitioning (cassandra 1.1.9). Any hint why this is happening and how to solve it? I would expect se= rver loads to be more equal along the cluster... Best regards, -- Marcelo Elias Del Valle http://mvalle.com - @mvallebr