incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeremiah D Jordan <jeremiah.jor...@gmail.com>
Subject Re: Cassandra 2.0 unbalanced ring with vnodes after adding new node
Date Mon, 09 Jun 2014 17:27:07 GMT
That looks like you started the initial nodes with num tokens=1, then later switched to vnodes,
by setting num tokens to 256, then added that new node with 256 vnodes to start.  Am I right?

Since you don't have very much data, the easiest way out of this will be to decommission the
original nodes one at a time.  Wipe all the data off.  Then bootstrap them back into the cluster.

-Jeremiah

On Jun 4, 2014, at 11:52 AM, Владимир Рудев <vladimir.rudev@gmail.com>
wrote:

> Hello to everyone!
> 
> Please, can someone explain where we made a mistake?
> 
> We have cluster with 4 nodes which uses vnodes(256 per node, default settings), snitch
is default on every node: SimpleSnitch.
> These four nodes was from beginning of a cluster.
> In this cluster we have keyspace with this options:
> Keyspace: K:
>   Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
>   Durable Writes: true
>     Options: [replication_factor:3]
> 
> All was normal and nodetool status K shows that each node owns 75% of all key range.
All 4 nodes are located in same datacenter and have same first two bytes in IP address(others
are different). 
> 
> Then we buy new server on different datacenter and add it to the cluster with same settings
as in previous four nodes(difference only in listen_address), assuming that the effective
own of each node for this keyspace will be 300/5=60% or near. But after 3-5 minutes after
start nodetool status K show this:
> nodetool status K;
> Datacenter: datacenter1
> =======================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address        Load       Tokens  Owns (effective)  Host ID                     
         Rack
> UN  N1   	6,06 GB    256     50.0%             62f295b3-0da6-4854-a53a-f03d6b424b03 
rack1
> UN  N2  	5,89 GB    256     50.0%             af4e4a23-2610-44dd-9061-09c7a6512a54  rack1
> UN  N3   	6,02 GB    256     50.0%             0f0e4e78-6fb2-479f-ad76-477006f76795 
rack1
> UN  N4   	5,8 GB     256     50.0%             670344c0-9856-48cf-9ec9-1a98f9a89460 
rack1
> UN  N5  	7,51 GB    256     100.0%            82473d14-9e36-4ae7-86d2-a3e526efb53f  rack1
> 
> N5 is newly added node
> 
> nodetool repair -pr on N5 doesn't change anything
> 
> nodetool describering K shows that new node N5 participate in EACH range. This is not
we want at all. 
> 
> It looks like cassandra add new node to each range because it located in different datacenter,
but all settings and output are exactly prevent this.
> 
> Also interesting point is that while in all config files snitch is defined as SimpleSnitch
the output of the command nodetool describecluster is:
> Cluster Information:
>         Name: Some Cluster Name
>         Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
>         Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>         Schema versions:
>                 26b8fa37-e666-31ed-aa3b-85be75f2aa1a: [N1, N2, N3, N4, N5]
> 
> We use Cassandra 2.0.6
> 
> Questions we have at this moment:
> 1. How to rebalance ring so all nodes will own 60% of range?
>    1a. Removing node from cluster and adding it again is a solution?
> 2. Where we possibly make a mistake when adding new node?
> 3. If we add new 6th node to ring it will take 50% from N5 or some portion from each
node?
> 
> Thanks in advance!
> 
> -- 
> С уважением, 
> Владимир Рудев
> (With regards, Vladimir Rudev)
> vladimir.rudev@gmail.com
> 
> 


Mime
View raw message