cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Colby <jonathan.co...@gmail.com>
Subject Re: fixing unbalanced cluster !?
Date Thu, 09 Jun 2011 15:19:48 GMT
Thanks Ben.   That's what I was afraid I had to do.  I can see how it's a lot easier if you
simply double the cluster when adding capacity.

Jon
 
On Jun 9, 2011, at 4:44 PM, Benjamin Coverston wrote:

> Because you were able to successfully run repair you can follow up with a nodetool cleanup
which will git rid of some of the extraneous data on that (bigger) node. You're also assured
after you run repair that entropy beteen the nodes is minimal.
> 
> Assuming you're using the random ordered partitioner: To balance your ring I would start
by calculating the new token locations, then moving each of your nodes backwards along their
owned range to their new locations.
> 
> From the script on http://wiki.apache.org/cassandra/Operations your new balanced tokens
would be:
> 
> 0
> 21267647932558653966460912964485513216
> 42535295865117307932921825928971026432
> 63802943797675961899382738893456539648
> 85070591730234615865843651857942052864
> 106338239662793269832304564822427566080
> 127605887595351923798765477786913079296
> 148873535527910577765226390751398592512
> 
> From this you can see that  10.46.108.{100, 101} is already in the right place so you
don't have to do anything with those nodes. Proceed with moving 10.46.108.104 to its new token,
the safest way to do this would be to use nodetool move. Another way to do it could be to
run a remove-token followed by re-adding the node into the ring at its new location. The risk
here is that if you do not at least repair after re-joining the ring (and before you move
the next node in the ring) then some of the data on that node would be ignored as it would
now fall out of the owned range, so it's good practice to immediately run repair on a node
that you do a removetoken / re-join on.
> 
> The rest of your balancing should be an iteration on the above steps moving through the
range.
> 
> 
> On 6/9/11 6:21 AM, Jonathan Colby wrote:
>> I got myself into a situation where one node (10.47.108.100) has a lot more data
than the other nodes.   In fact, the 1 TB disk on this node is almost full.  I added 3 new
nodes and let cassandra automatically calculate new tokens by taking the highest loaded nodes.
 Unfortunately there is still a big token range this  node is responsible for (5113... - 
85070...).  Yes, I know that one option would be to rebalance the entire cluster with move
but this is an extremely time-consuming and error-prone process because of the amount of data
involved.
>> 
>> Our RF = 3 and we read/write quorum.   The nodes have been repaired so I think the
data should be in good shape.
>> 
>> Question:    Can I get myself out of this mess without installing new nodes?    I
was thinking of either decommission or removetoken to have the cluster "rebalance itself".
 The re-bootstrap this node to a new token.
>> 
>> 
>> Address         Status State   Load            Owns    Token
>>                                                        127605887595351923798765477786913079296
>> 10.46.108.100   Up     Normal  218.52 GB       25.00%  0
>> 10.46.108.101   Up     Normal  260.04 GB       12.50%  21267647932558653966460912964485513216
>> 10.46.108.104   Up     Normal  286.79 GB       17.56%  51138582157040063602728874106478613120
>> 10.47.108.100   Up     Normal  874.91 GB       19.94%  85070591730234615865843651857942052863
>> 10.47.108.102   Up     Normal  302.79 GB       4.16%   92156241323118845370666296304459139297
>> 10.47.108.103   Up     Normal  242.02 GB       4.16%   99241191538897700272878550821956884116
>> 10.47.108.101   Up     Normal  439.9 GB        8.34%   113427455640312821154458202477256070484
>> 10.46.108.103   Up     Normal  304 GB          8.33%   127605887595351923798765477786913079296
> 
> -- 
> Ben Coverston
> Director of Operations
> DataStax -- The Apache Cassandra Company
> http://www.datastax.com/
> 


Mime
View raw message