cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Piavlo <lolitus...@gmail.com>
Subject Re: strange gossip messages after node reboot with different ip
Date Tue, 08 May 2012 07:12:39 GMT
On 05/01/2012 04:16 AM, aaron morton wrote:
> Gossip information about a node can stay in the cluster for up to 3 
> days. How long has this been going on for ?
This has been going for over a week already without any signs of slow 
down, all nodes that have changed ip popup as UP/DEAD endlessly.
Any ideas?

Thanks
>
> I'm unsure if this is expected behaviour. But it sounds like Gossip is 
> kicking out the phantom node correctly.
>
> Can you use nodetool gossipinfo on the nodes to capture some artefacts 
> while it is still running?
>
>> How come the old ip 10.63.14.214 still popup as UP and then declared 
>> as DEAD again, an so on and on?
> I think this is gossip bouncing information about the node around. 
> Once it has been observed as dead for 3 days it should be purged.
>> Another question, if node is recognised as new (due to ip change) but 
>> with same token - will other nodes stream the hinted handoffs to it?
> Hints are stored against the token, not the end point address. When a 
> node comes up the process is reversed and the end point is mapped to 
> it's (new) token.
>
>>  And is there way to tell cassandra also use names and if ip changes 
>> but node name is the same and resolves to the new ip then the cluster 
>> treat it as old node?
>>
> Not that I am aware of. It's designed to handle IP addresses changing. 
> AFAIK the log messages are not indicative of a fault. Instead they 
> indicate something odd happening with Gossip that is being correctly 
> handled.
>
> Hope that helps.
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 1/05/2012, at 3:09 AM, Piavlo wrote:
>
>>
>> Hi,
>>
>> We have a cassandra cluster in ec2.
>> If i stop a node and start it - as a result the node ip changes. The 
>> node is recognised as NEW node and is declared as replacing the 
>> previous node with same token.(But this is the same node of course)
>>
>> In this specific case the node ip before stop/start was 10.63.14.214 
>> and new ip is 10.54.81.14.
>> And even that the cluster and node  seems to be working fine for more 
>> than a day after the stop/start of this node, I see the following 
>> loop of messages ~ once every minute.
>>
>> INFO [GossipStage:1] 2012-04-30 14:18:57,089 Gossiper.java (line 838) 
>> Node /10.63.14.214 is now part of the cluster
>> INFO [GossipStage:1] 2012-04-30 14:18:57,089 Gossiper.java (line 804) 
>> InetAddress /10.63.14.214 is now UP
>> INFO [GossipStage:1] 2012-04-30 14:18:57,090 StorageService.java 
>> (line 1017) Nodes /10.63.14.214 and cassa1a.internal/10.54.81.14 have 
>> the same token 0.  Ignoring /10.63.14.214
>> INFO [GossipTasks:1] 2012-04-30 14:19:11,834 Gossiper.java (line 818) 
>> InetAddress /10.63.14.214 is now dead.
>> INFO [GossipTasks:1] 2012-04-30 14:19:27,896 Gossiper.java (line 632) 
>> FatClient /10.63.14.214 has been silent for 30000ms, removing from gossip
>> INFO [GossipStage:1] 2012-04-30 14:20:30,803 Gossiper.java (line 838) 
>> Node /10.63.14.214 is now part of the cluster
>> ...
>>
>> How come the old ip 10.63.14.214 still popup as UP and then declared 
>> as DEAD again, an so on and on?
>> I know since this is ec2 other node with same ip can come UP, but 
>> i've verified and there is no such node and it certainly does not run 
>> cassandra :)
>> I stop/started another node and observe similar behaviour.
>> This is version 1.0.8
>>
>> Another question, if node is recognised as new (due to ip change) but 
>> with same token - will other nodes stream the hinted handoffs to it?
>> And is there way to tell cassandra also use names and if ip changes 
>> but node name is the same and resolves to the new ip then the cluster 
>> treat it as old node?
>>
>> Thanks
>> Alex
>


Mime
View raw message