incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Colby <jonathan.co...@gmail.com>
Subject fixing unbalanced cluster !?
Date Thu, 09 Jun 2011 12:21:51 GMT
I got myself into a situation where one node (10.47.108.100) has a lot more data than the other
nodes.   In fact, the 1 TB disk on this node is almost full.  I added 3 new nodes and let
cassandra automatically calculate new tokens by taking the highest loaded nodes.  Unfortunately
there is still a big token range this  node is responsible for (5113... -  85070...).  Yes,
I know that one option would be to rebalance the entire cluster with move but this is an extremely
time-consuming and error-prone process because of the amount of data involved.  

Our RF = 3 and we read/write quorum.   The nodes have been repaired so I think the data should
be in good shape.

Question:    Can I get myself out of this mess without installing new nodes?    I was thinking
of either decommission or removetoken to have the cluster "rebalance itself".  The re-bootstrap
this node to a new token.


Address         Status State   Load            Owns    Token                             
         
                                                       127605887595351923798765477786913079296
    
10.46.108.100   Up     Normal  218.52 GB       25.00%  0                                 
         
10.46.108.101   Up     Normal  260.04 GB       12.50%  21267647932558653966460912964485513216
     
10.46.108.104   Up     Normal  286.79 GB       17.56%  51138582157040063602728874106478613120
     
10.47.108.100   Up     Normal  874.91 GB       19.94%  85070591730234615865843651857942052863
     
10.47.108.102   Up     Normal  302.79 GB       4.16%   92156241323118845370666296304459139297
     
10.47.108.103   Up     Normal  242.02 GB       4.16%   99241191538897700272878550821956884116
     
10.47.108.101   Up     Normal  439.9 GB        8.34%   113427455640312821154458202477256070484
    
10.46.108.103   Up     Normal  304 GB          8.33%   127605887595351923798765477786913079296
 
Mime
View raw message