incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andreas Finke <Andreas.Fi...@solvians.com>
Subject Unbalanced ring with C* 2.0.3 and vnodes after adding additional nodes
Date Thu, 12 Dec 2013 10:49:46 GMT
Hi,

after adding 2 more nodes to a 4 nodes cluster (before) we are experiencing high load on both
new nodes. After doing some investigation we found out the following:

- High cpu load on vm5+6
- Higher data load on vm5+6
- Write requests are evenly distributed to all 6 nodes by our client application (opscenter
-> metrics -> WriteRequests)
- Local writes are as twice as much in vm5 +6 (vm1-4: ~2800/s, vm5-6: ~6800/s)
- Nodetool output:


UN  vm1  9.51 GB    256     20,7%  13fa7bb7-19cb-44f5-af83-71a72e04993a  X1

UN  vm2  9.41 GB    256     20,0%  b71c2d3d-4721-4dde-a418-802f1af4b7a1  D1

UN  vm3  9.37 GB    256     18,9%  8ce4c419-d79c-4ef1-b3fd-8936bff3e44f  X1

UN  vm4  9.23 GB    256     19,5%  17974f20-5756-4eba-a377-52feed3a1b10  D1

UN  vm5  15.95 GB   256     10,7%  0c6db9ea-4c60-43f6-a12e-51a7d76f8e80  X1

UN  vm6  14.86 GB   256     10,2%  f64d1909-dd96-442b-b602-efee29eee0a0  D1


Although the ownership is lower on vm5-6 (which already is not right) the data load is way
higher.


Some cluster facts:


Node: 4 CPU, 6 GB RAM, virtual appliance

Cassandra: 3 GB Heap, vnodes 256

Schema: Replication strategy network, RF:2


Has anyone an idea what could be the cause for the unbalancing. Maybe we forgot necessary
actions during or after cluster expanding process. We are open for every idea.


Regards

Andi



Mime
View raw message