cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shiwen Cheng <>
Subject Issue with removing a node and adding it back
Date Thu, 26 Mar 2015 18:31:13 GMT
Hi all,

I encountered an issue by removing and adding back a node.
Here is how this issue came out:
(1) We have four nodes cluster running, but there was a hard disk failure
on one of the node.
Since we need to replace the hard disk, I chose to use *removenode *to
remove the failed node.
(2) few days later, after the new hard disk is installed. I re-install the
Cassandra on this node.
I checked the .yaml file and it is the same as the other three nodes (only
difference is the listen_address), the newly added node is not in the seeds
I used the same Cassandra version as other nodes which is 2.0.5
(3) With bootstrap set to true (by default), the new node "seems" can join
the cluster.
 (a) OpsCenter shows this node is with "*unknow datacenter*".
 (b) the status of this node in OpsCenter is shown as "*joining*"
 (c) one of the node starts to streaming the data to the new node.
However, after few hours there is no futher streaming, but the data size is
not even close to other nodes which is definitely not finished.
 (d) The node is still shown as "joining" with "unknown datacenter"  in
OpsCenter.  More than 12 hours in this status.
 (e) *nodetool status* on other three machines doesn't show this newly
added node.

There are no exceptions in the log from the newly added node.
I tried many times to re-install cassandra, opscenter and datastax-agent
but no luck to solve it.
So I got stuck here.

Can anybody help? I really appreciate!


View raw message