incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Morton <aa...@thelastpickle.com>
Subject Re: Unbalanced ring with C* 2.0.3 and vnodes after adding additional nodes
Date Wed, 18 Dec 2013 02:14:57 GMT
> Node: 4 CPU, 6 GB RAM, virtual appliance
> 
> Cassandra: 3 GB Heap, vnodes 256
FWIW that’s a very low powered node. 

> Maybe we forgot necessary actions during or after cluster expanding process. We are open
for every idea.
Where the nodes in the seed list when they joined the cluster? If so they do not bootstrap.


The extra writes in nodes 5 and 6 could be from Read Repair writing to them. 

Cheers

-----------------
Aaron Morton
New Zealand
@aaronmorton

Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

On 12/12/2013, at 11:49 pm, Andreas Finke <Andreas.Finke@solvians.com> wrote:

> Hi,
> 
> after adding 2 more nodes to a 4 nodes cluster (before) we are experiencing high load
on both new nodes. After doing some investigation we found out the following:
> 
> - High cpu load on vm5+6
> - Higher data load on vm5+6
> - Write requests are evenly distributed to all 6 nodes by our client application (opscenter
-> metrics -> WriteRequests) 
> - Local writes are as twice as much in vm5 +6 (vm1-4: ~2800/s, vm5-6: ~6800/s)
> - Nodetool output:
> 
> UN  vm1  9.51 GB    256     20,7%  13fa7bb7-19cb-44f5-af83-71a72e04993a  X1
> 
> UN  vm2  9.41 GB    256     20,0%  b71c2d3d-4721-4dde-a418-802f1af4b7a1  D1
> 
> UN  vm3  9.37 GB    256     18,9%  8ce4c419-d79c-4ef1-b3fd-8936bff3e44f  X1
> 
> 
> UN  vm4  9.23 GB    256     19,5%  17974f20-5756-4eba-a377-52feed3a1b10  D1
> 
> UN  vm5  15.95 GB   256     10,7%  0c6db9ea-4c60-43f6-a12e-51a7d76f8e80  X1
> 
> UN  vm6  14.86 GB   256     10,2%  f64d1909-dd96-442b-b602-efee29eee0a0  D1
> 
> 
> 
> Although the ownership is lower on vm5-6 (which already is not right) the data load is
way higher. 
> 
> 
> 
> Some cluster facts:
> 
> 
> 
> Node: 4 CPU, 6 GB RAM, virtual appliance
> 
> Cassandra: 3 GB Heap, vnodes 256
> 
> Schema: Replication strategy network, RF:2
> 
> 
> 
> Has anyone an idea what could be the cause for the unbalancing. Maybe we forgot necessary
actions during or after cluster expanding process. We are open for every idea.
> 
> 
> 
> Regards
> 
> Andi
> 


Mime
View raw message