cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Questions about the nodetool ring.
Date Tue, 12 Apr 2011 12:01:41 GMT
If you are seeing a different views of the ring from different nodes you may have some sickness

http://www.datastax.com/docs/0.7/troubleshooting/index#view-of-ring-differs-between-some-nodes

The "?" in the ring output happens when one node does not know if the other is alice or dead.
This could be due to the corrupt gossip state described in the link above. 

During a move the node will decommission and stop taking requests for the rang it was responsible
for. But other nodes int he cluster will take it's place. Once it starts bootstrapping it
will start accepting writes but not reads. The cluster stays online for all token ranges.

Dikang, did you allow the first move to complete before starting the second?

Aaron
  	
On 12 Apr 2011, at 22:55, Jonathan Colby wrote:

> when you do a move, the node is decommissioned and bootstrapped. During the autobootstrap
process the node will not receive reads until bootstrapping is complete.  I assume during
the decommission phase the node will also be unavailable,  someone correct me if I'm wrong.
> 
> the ring distribution looks better now.
> 
> The "?" I get all the time too.   And if you run "ring" against different hosts, the
question marks probably appear in different places.   I'm not sure if it means there is a
problem.  I haven't taken those question marks too seriously.
> 
> 
> 
> On Apr 12, 2011, at 11:57 AM, Dikang Gu wrote:
> 
>> After the nodetool move, I got this:
>> 
>> [root@server3 apache-cassandra-0.7.4]# bin/nodetool -h 10.18.101.213 ring
>> Address         Status State   Load            Owns    Token                    
                  
>>                                                        113427455640312821154458202477256070485
    
>> 10.18.101.211   ?      Normal  82.31 MB        33.33%  0                        
                  
>> 10.18.101.212   ?      Normal  84.24 MB        33.33%  56713727820156410577229101238628035242
     
>> 10.18.101.213   Up     Normal  54.44 MB        33.33%  113427455640312821154458202477256070485
>> 
>> Is this correct? Why is the status "?" ?
>> 
>> Thanks.
>> 
>> On Tue, Apr 12, 2011 at 5:43 PM, Dikang Gu <dikang85@gmail.com> wrote:
>> The 3 nodes were added to the cluster at the same time, so I'm not sure whey the
data vary.
>> 
>> I calculate the tokens and get:
>> node 0: 0
>> node 1: 56713727820156410577229101238628035242
>> node 2: 113427455640312821154458202477256070485
>> 
>> So I should set these tokens to the three nodes?  
>> 
>> And during the time I execute the nodetool move commands, can the cassandra servers
serve the front end requests at the same time? Is the data safe?
>> 
>> Thanks.
>> 
>> On Tue, Apr 12, 2011 at 5:15 PM, Jonathan Colby <jonathan.colby@gmail.com>
wrote:
>> This is "normal" when you just add single nodes.   When no token is assigned, the
new node takes a portion of the ring from the most heavily loaded node.    As a consequence
of this, the nodes will be out of balance.
>> 
>> In other words, when you double the amount nodes you would not have this problem.
>> 
>> The best way to rebalance the cluster is to generate new tokens and use the nodetool
move <new-token> command to rebalance the nodes, one at a time.
>> 
>> After rebalancing you can run "cleanup" so the nodes get rid of data they no longer
are responsible for.
>> 
>> links:
>> 
>> http://wiki.apache.org/cassandra/Operations#Range_changes
>> 
>> http://wiki.apache.org/cassandra/Operations#Moving_or_Removing_nodes
>> 
>> http://www.datastax.com/docs/0.7/operations/clustering#adding-capacity
>> 
>> 
>> 
>> On Apr 12, 2011, at 11:00 AM, Dikang Gu wrote:
>> 
>> > I have 3 cassandra 0.7.4 nodes in a cluster, and I get the ring stats:
>> >
>> > [root@yun-phy2 apache-cassandra-0.7.4]# bin/nodetool -h 192.168.1.28 -p 8090
ring
>> > Address         Status State   Load            Owns    Token
>> >                                                        109028275973926493413574716008500203721
>> > 192.168.1.25    Up     Normal  157.25 MB       69.92%  57856537434773737201679995572503935972
>> > 192.168.1.27    Up     Normal  201.71 MB       24.28%  99165710459060760249270263771474737125
>> > 192.168.1.28    Up     Normal  68.12 MB        5.80%   109028275973926493413574716008500203721
>> >
>> > The load and owns vary on each node, is this normal?  And is there a way to
balance the three nodes?
>> >
>> > Thanks.
>> >
>> > --
>> > Dikang Gu
>> >
>> > 0086 - 18611140205
>> >
>> 
>> 
>> 
>> 
>> -- 
>> Dikang Gu
>> 
>> 0086 - 18611140205
>> 
>> 
>> 
>> 
>> -- 
>> Dikang Gu
>> 
>> 0086 - 18611140205
>> 
> 


Mime
View raw message