incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hiller, Dean" <Dean.Hil...@nrel.gov>
Subject Re: load on cluster nodes
Date Wed, 20 Feb 2013 23:07:41 GMT
Check the logs for compaction.  CPU will go up while one node may be compacting and the other
node is not compacting yet.  Compaction can also last quite a long time in some cases.  Also,
you can do a jstack –l {pid} > thread.txt to do a thread dump to see what that node is
doing and compare it with other nodes.

I am actually in the process of doing quite a bit of performance analysis myself and I know
I need to go back and tweak the astyanax client somehow as well as I found I can run a test
of running jmap heap dump on node 1 in my cluster and our website sometimes slows down(ie.
I keep refreshing the pages) and I feel astyanax is going back to node 1 too often which results
in a 10 second page load vs. when it hits nodes that are fine, it loads in 300ms.

We have been running live for a while and get various percentage bumps on some nodes vs. others
but mostly has been compaction running at different times for quite a while.

Dean

From: Marcelo Elias Del Valle <mvallebr@gmail.com<mailto:mvallebr@gmail.com>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Date: Wednesday, February 20, 2013 3:53 PM
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: load on cluster nodes

Hello,

     I am writing a lot of data on a cassandra cluster with 3 nodes. All my three nodes are
seeds, when I create the astyanax connection pool I specify ip1,ip2,ip3 as seeds and so in
the casandra.yaml config file.
      However, if I look at the server load when I am bombarding it with data, I can see the
first server is clearly being more demanded (graph here: http://mvalle.com/downloads/serverload1.png)
. I am writting data in sequence, generating UUIDs for column families based on timestamp.
Also, I am using default partitioning (cassandra 1.1.9).
      Any hint why this is happening and how to solve it? I would expect server loads to be
more equal along the cluster...

Best regards,
--
Marcelo Elias Del Valle
http://mvalle.com - @mvallebr

Mime
View raw message