incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas van Neerijnen <...@bossastudios.com>
Subject Re: replaced node keeps returning in gossip
Date Fri, 19 Oct 2012 11:25:51 GMT
Hi

When I sent the mail I'd had the new node on for about an hour, the old
node died about an hour before that.
The weirdness in the log files stopped yesterday afternoon, about 4 or 5
hours after I replaced the node so it seems to have resolved itself.
Seeing as there's no error to look at in the log files not sure if you
still want the output of my gossipinfo but I've pasted it below anyway.
Thanks!

/10.16.96.212
  LOAD:7.8018521345E10
  RPC_ADDRESS:0.0.0.0
  RELEASE_VERSION:1.0.11
  STATUS:NORMAL,113427455640312821154458202477256070484
  SCHEMA:9b152e00-fd90-11e1-0000-2d22988ca597
/10.16.128.211
  LOAD:7.8416250275E10
  RPC_ADDRESS:0.0.0.0
  RELEASE_VERSION:1.0.11
  STATUS:NORMAL,85070591730234615865843651857942052863
  SCHEMA:9b152e00-fd90-11e1-0000-2d22988ca597
/10.16.32.210
  LOAD:1.29054735121E11
  RPC_ADDRESS:0.0.0.0
  RELEASE_VERSION:1.0.11
  STATUS:NORMAL,56713727820156407428984779325531226112
  SCHEMA:9b152e00-fd90-11e1-0000-2d22988ca597
/10.16.32.211
  LOAD:7.2937725831E10
  RPC_ADDRESS:0.0.0.0
  RELEASE_VERSION:1.0.11
  STATUS:NORMAL,141784319550391032739561396922763706368
  SCHEMA:9b152e00-fd90-11e1-0000-2d22988ca597
ip-10-16-128-197.localdomain/10.16.128.197
  LOAD:6.5571879526E10
  RPC_ADDRESS:0.0.0.0
  RELEASE_VERSION:1.0.11
  STATUS:NORMAL,0
  SCHEMA:9b152e00-fd90-11e1-0000-2d22988ca597
/10.16.96.211
  LOAD:1.0633383453E11
  RPC_ADDRESS:0.0.0.0
  RELEASE_VERSION:1.0.11
  STATUS:NORMAL,28356863910078203714492389662765613056
  SCHEMA:9b152e00-fd90-11e1-0000-2d22988ca597


On Fri, Oct 19, 2012 at 2:56 AM, aaron morton <aaron@thelastpickle.com>wrote:

> I replaced it with a new node, IP 10.16.128.197 and again token 0 with a
> "-Dcassandra.replace_token=0" at startup
>
> Good Good.
>
> How long ago did you bring the new node on ? There is a fail safe to
> remove 128.210 after 3 days if it does not gossip to other nodes.
>
> I *thought* that remove_token would remove the old IP from the ring. Can
> you post the output from nodetool gossipinfo from the 128.197 node ?
>
> Thanks
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 19/10/2012, at 2:44 AM, Thomas van Neerijnen <tom@bossastudios.com>
> wrote:
>
> Hi all
>
> I'm running Cassandra 1.0.11 on Ubuntu 11.10.
>
> I've got a ghost node which keeps showing up on my ring.
>
> A node living on IP 10.16.128.210 and token 0 died and had to be replaced.
> I replaced it with a new node, IP 10.16.128.197 and again token 0 with a
> "-Dcassandra.replace_token=0" at startup. This all went well but now I'm
> seeing the following weirdness constantly reported in the log files around
> the ring:
>
>  INFO [GossipTasks:1] 2012-10-18 13:39:22,441 Gossiper.java (line 632)
> FatClient /10.16.128.210 has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-10-18 13:40:25,933 Gossiper.java (line 838)
> Node /10.16.128.210 is now part of the cluster
>  INFO [GossipStage:1] 2012-10-18 13:40:25,934 Gossiper.java (line 804)
> InetAddress /10.16.128.210 is now UP
>  INFO [GossipStage:1] 2012-10-18 13:40:25,937 StorageService.java (line
> 1017) Nodes /10.16.128.210 and /10.16.128.197 have the same token 0.
> Ignoring /10.16.128.210
>  INFO [GossipTasks:1] 2012-10-18 13:40:37,509 Gossiper.java (line 818)
> InetAddress /10.16.128.210 is now dead.
>  INFO [GossipTasks:1] 2012-10-18 13:40:56,526 Gossiper.java (line 632)
> FatClient /10.16.128.210 has been silent for 30000ms, removing from gossip
>
>
>

Mime
View raw message