cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ahamed, Aadil" <>
Subject Effect of adding/removing nodes from cassandra
Date Mon, 31 Aug 2015 18:19:31 GMT

This is my first post to this mailing list so I want to apologize in advance if I break any
rules or guidelines. I have also inserted images of graphs and I am not sure if they will
show up. Please let me know if I can improve my post in any way.

I am investigating the effect of adding and removing nodes from a Cassandra cluster by collecting
metrics on cpu utilization, memory usage etc.
Basically I want to have a quantitative measurement of how intensive the add/remove operation
is so we know what to expect when increasing/decreasing capacity on our production cluster.
I wrote a tool to measure this effect but I need help interpreting the data I have collected.

Here is a sample testcase:

Relevant Machine Information:
Logical cores: 8
Core model: Intel Xeon(R) CPU E31270 @ 3.4GHz
Non-Volatile storage: Hard Disk

Test Info:
Keyspace: Simple strategy with replication factor 2
Initial no. of nodes on cluster: 2 [,]
Initial load on each node: 30GB
No. of added nodes: 1 []
Final load on each node: 18GB

*The x-axis for all graphs is in seconds

Graph for initial node 1:


Graph for initial node 2:


Graph for added node:

Here are my questions:

Graph for initial node 1, 2:

1. Why is the KBytes/s sent over the network stay constant at around ~5000 KB/s? Can I change
it to something higher?

Graph for added node:

1.  Why is the cpu usage so high?
It is close to 30% for ~800 seconds and 10% for another 400 seconds. I did not expect the
cpu usage to be this high and I did not expect it to stay high for such a long period of time.
Based on my understanding the only cpu intensive process during bootstrap is token range recalculation.
However, based on the graph the cpu usage seems to be proportional to the amount of data that
is streamed to the node at any given moment in time.

2. Why does the Kbytes/s received over the network drop from - ~10000 -> ~5000  - at around
800 seconds?
I can see that initial node 1 stops streaming data at around 800 seconds but why does initial
node 2 not bump up its outgoing rate of transfer?

Thanks for your help,

View raw message