cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: strange gossip messages after node reboot with different ip
Date Tue, 01 May 2012 01:16:29 GMT
Gossip information about a node can stay in the cluster for up to 3 days. How long has this
been going on for ? 

I'm unsure if this is expected behaviour. But it sounds like Gossip is kicking out the phantom
node correctly.

Can you use nodetool gossipinfo on the nodes to capture some artefacts while it is still running?

> How come the old ip 10.63.14.214 still popup as UP and then declared as DEAD again, an
so on and on?
I think this is gossip bouncing information about the node around. Once it has been observed
as dead for 3 days it should be purged.
  
> Another question, if node is recognised as new (due to ip change) but with same token
- will other nodes stream the hinted handoffs to it?
Hints are stored against the token, not the end point address. When a node comes up the process
is reversed and the end point is mapped to it's (new) token.

>  And is there way to tell cassandra also use names and if ip changes but node name is
the same and resolves to the new ip then the cluster treat it as old node?
> 
Not that I am aware of. It's designed to handle IP addresses changing. AFAIK the log messages
are not indicative of a fault. Instead they indicate something odd happening with Gossip that
is being correctly handled. 

Hope that helps. 
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 1/05/2012, at 3:09 AM, Piavlo wrote:

> 
> Hi,
> 
> We have a cassandra cluster in ec2.
> If i stop a node and start it - as a result the node ip changes. The node is recognised
as NEW node and is declared as replacing the previous node with same token.(But this is the
same node of course)
> 
> In this specific case the node ip before stop/start was 10.63.14.214 and new ip is 10.54.81.14.
> And even that the cluster and node  seems to be working fine for more than a day after
the stop/start of this node, I see the following loop of messages ~ once every minute.
> 
> INFO [GossipStage:1] 2012-04-30 14:18:57,089 Gossiper.java (line 838) Node /10.63.14.214
is now part of the cluster
> INFO [GossipStage:1] 2012-04-30 14:18:57,089 Gossiper.java (line 804) InetAddress /10.63.14.214
is now UP
> INFO [GossipStage:1] 2012-04-30 14:18:57,090 StorageService.java (line 1017) Nodes /10.63.14.214
and cassa1a.internal/10.54.81.14 have the same token 0.  Ignoring /10.63.14.214
> INFO [GossipTasks:1] 2012-04-30 14:19:11,834 Gossiper.java (line 818) InetAddress /10.63.14.214
is now dead.
> INFO [GossipTasks:1] 2012-04-30 14:19:27,896 Gossiper.java (line 632) FatClient /10.63.14.214
has been silent for 30000ms, removing from gossip
> INFO [GossipStage:1] 2012-04-30 14:20:30,803 Gossiper.java (line 838) Node /10.63.14.214
is now part of the cluster
> ...
> 
> How come the old ip 10.63.14.214 still popup as UP and then declared as DEAD again, an
so on and on?
> I know since this is ec2 other node with same ip can come UP, but i've verified and there
is no such node and it certainly does not run cassandra :)
> I stop/started another node and observe similar behaviour.
> This is version 1.0.8
> 
> Another question, if node is recognised as new (due to ip change) but with same token
- will other nodes stream the hinted handoffs to it?
> And is there way to tell cassandra also use names and if ip changes but node name is
the same and resolves to the new ip then the cluster treat it as old node?
> 
> Thanks
> Alex


Mime
View raw message