cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Piavlo <>
Subject strange gossip messages after node reboot with different ip
Date Mon, 30 Apr 2012 15:09:24 GMT


We have a cassandra cluster in ec2.
If i stop a node and start it - as a result the node ip changes. The 
node is recognised as NEW node and is declared as replacing the previous 
node with same token.(But this is the same node of course)

In this specific case the node ip before stop/start was and 
new ip is
And even that the cluster and node  seems to be working fine for more 
than a day after the stop/start of this node, I see the following loop 
of messages ~ once every minute.

  INFO [GossipStage:1] 2012-04-30 14:18:57,089 (line 838) 
Node / is now part of the cluster
  INFO [GossipStage:1] 2012-04-30 14:18:57,089 (line 804) 
InetAddress / is now UP
  INFO [GossipStage:1] 2012-04-30 14:18:57,090 (line 
1017) Nodes / and cassa1a.internal/ have the same 
token 0.  Ignoring /
  INFO [GossipTasks:1] 2012-04-30 14:19:11,834 (line 818) 
InetAddress / is now dead.
  INFO [GossipTasks:1] 2012-04-30 14:19:27,896 (line 632) 
FatClient / has been silent for 30000ms, removing from gossip
  INFO [GossipStage:1] 2012-04-30 14:20:30,803 (line 838) 
Node / is now part of the cluster

How come the old ip still popup as UP and then declared as 
DEAD again, an so on and on?
I know since this is ec2 other node with same ip can come UP, but i've 
verified and there is no such node and it certainly does not run 
cassandra :)
I stop/started another node and observe similar behaviour.
This is version 1.0.8

Another question, if node is recognised as new (due to ip change) but 
with same token - will other nodes stream the hinted handoffs to it?
And is there way to tell cassandra also use names and if ip changes but 
node name is the same and resolves to the new ip then the cluster treat 
it as old node?


View raw message