incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andreas Finke <Andreas.Fi...@solvians.com>
Subject RE: Unbalanced ring with C* 2.0.3 and vnodes after adding additional nodes
Date Fri, 20 Dec 2013 09:51:22 GMT
Hi Aaron,

I assume you mean "seed_provider" setting in cassandra.yaml by "seed list". The current setting
for vm1-vm6 is:

seed_provider = vm1,vm2,vm3,vm4

This setting also applied when the vm5 and vm6 were added. I checked the read repair metrics
and it is about mean 20/s on vm5 and vm6. 

I tried to investigate the real distribution of tokens again and did on vm1:

1. nodetool describering marketdata >> /tmp/ring.txt
2. for node in vm1 vm2 vm3 vm4 vm5 vm6 ; do cat /tmp/ring.txt |grep <ip_of($node)> |
wc -l; done

This prints the number of times when a node was listed as endpoint:

vm1: 303
vm2: 312
vm3: 332
vm4: 311
vm5: 901
vm6: 913

So this shows that we are really unbalanced. 

1. Is there any way how we can fix that on a running production cluster?
2. Our backup plan is to snapshot all data, raise a complete fresh 6 node cluster and stream
the data using sstable loader. Are there any objections about that plan from your point of
view?

Thanks in advance!

Andi
________________________________________
From: Aaron Morton [aaron@thelastpickle.com]
Sent: Wednesday, December 18, 2013 3:14 AM
To: Cassandra User
Subject: Re: Unbalanced ring with C* 2.0.3 and vnodes after adding additional nodes

> Node: 4 CPU, 6 GB RAM, virtual appliance
>
> Cassandra: 3 GB Heap, vnodes 256
FWIW that’s a very low powered node.

> Maybe we forgot necessary actions during or after cluster expanding process. We are open
for every idea.
Where the nodes in the seed list when they joined the cluster? If so they do not bootstrap.

The extra writes in nodes 5 and 6 could be from Read Repair writing to them.

Cheers

-----------------
Aaron Morton
New Zealand
@aaronmorton

Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

On 12/12/2013, at 11:49 pm, Andreas Finke <Andreas.Finke@solvians.com> wrote:

> Hi,
>
> after adding 2 more nodes to a 4 nodes cluster (before) we are experiencing high load
on both new nodes. After doing some investigation we found out the following:
>
> - High cpu load on vm5+6
> - Higher data load on vm5+6
> - Write requests are evenly distributed to all 6 nodes by our client application (opscenter
-> metrics -> WriteRequests)
> - Local writes are as twice as much in vm5 +6 (vm1-4: ~2800/s, vm5-6: ~6800/s)
> - Nodetool output:
>
> UN  vm1  9.51 GB    256     20,7%  13fa7bb7-19cb-44f5-af83-71a72e04993a  X1
>
> UN  vm2  9.41 GB    256     20,0%  b71c2d3d-4721-4dde-a418-802f1af4b7a1  D1
>
> UN  vm3  9.37 GB    256     18,9%  8ce4c419-d79c-4ef1-b3fd-8936bff3e44f  X1
>
>
> UN  vm4  9.23 GB    256     19,5%  17974f20-5756-4eba-a377-52feed3a1b10  D1
>
> UN  vm5  15.95 GB   256     10,7%  0c6db9ea-4c60-43f6-a12e-51a7d76f8e80  X1
>
> UN  vm6  14.86 GB   256     10,2%  f64d1909-dd96-442b-b602-efee29eee0a0  D1
>
>
>
> Although the ownership is lower on vm5-6 (which already is not right) the data load is
way higher.
>
>
>
> Some cluster facts:
>
>
>
> Node: 4 CPU, 6 GB RAM, virtual appliance
>
> Cassandra: 3 GB Heap, vnodes 256
>
> Schema: Replication strategy network, RF:2
>
>
>
> Has anyone an idea what could be the cause for the unbalancing. Maybe we forgot necessary
actions during or after cluster expanding process. We are open for every idea.
>
>
>
> Regards
>
> Andi
>


Mime
View raw message