incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Richard Low <>
Subject Re: Problems with shuffle
Date Mon, 15 Apr 2013 14:10:25 GMT
On 14 April 2013 00:56, Rustam Aliyev <> wrote:

>  Just a followup on this issue. Due to the cost of shuffle, we decided
> not to do it. Recently, we added new node and ended up in not well balanced
> cluster:
> Datacenter: datacenter1
> =======================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address           Load       Tokens  Owns   Host
> ID                               Rack
> UN          52.28 GB   260     18.3%
> d28df6a6-c888-4658-9be1-f9e286368dce  rack1
> UN         55.21 GB   256     9.4%
> 7b0cf3c8-0c42-4443-9b0c-68f794299443  rack1
> UN          49.03 GB   259     17.9%
> 2d308bc3-1fd7-4fa4-b33f-cbbbdc557b2f  rack1
> UN          48.51 GB   255     18.4%
> c253dcdf-3e93-495c-baf1-e4d2a033bce3  rack1
> UN          67.14 GB   253     17.9%
> 4f77fd70-b134-486b-9c25-cfea96b6d412  rack1
> UN          47.65 GB   253     18.0%
> 4d03690d-5363-42c1-85c2-5084596e09fc  rack1
> It looks like new node took from each other node equal amount of vnodes -
> which is good. However, it's not clear why it decided to have twice less
> than other nodes.

I think this is expected behaviour when adding a node to a cluster that has
been upgraded to vnodes without shuffling.  The old nodes have equally
spaced contiguous tokens.  The new node will choose 256 random new tokens,
which will on average bisect the old ranges.  This means each token the new
node has will only cover half the range (on average) as the old ones.

However, the thing that really matters is the load, which is surprisingly
balanced at 55 GB.  This isn't guaranteed though - it could be about half
or it could be significantly more.  The problem with not doing the shuffle
is the vnode after all the contiguous vnodes for a certain node will be the
target for the second replica of *all* the vnodes for that node.  E.g. if
node A has tokens 10, 20, 30, 40, node B has tokens 50, 60, 70, 80 and node
C (the new node) chooses token 45, it will store a replica for all data
stored in A's tokens.  This is exactly the same reason as why tokens in a
multi-DC deployment need to be interleaved rather than be contiguous.

If shuffle isn't going to work, you could instead decommission each node
then bootstrap it in again.  In principle that should copy your data twice
as much as required (shuffle is optimal in terms of data transfer) but some
implementation details might make it more efficient.


View raw message