cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vijay (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-3736) -Dreplace_token leaves old node (IP) in the gossip with the token.
Date Thu, 19 Jan 2012 04:34:40 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188911#comment-13188911
] 

Vijay commented on CASSANDRA-3736:
----------------------------------

Yes and the fix attached with this ticket will also remove the node from the System table,
while replacing hence you wont even see the following message...

>>> INFO [GossipStage:1] 2012-01-18 23:45:56,412 Gossiper.java (line 800) InetAddress
/50.56.58.55 is now UP

The problem is that we remove the node after 30 seconds.... Meanwhile the gossip will make
the other node know about .55 and hence the message in the other node. 
The patch will fix this by removing the information from the System table in the first place
instead of restart which triggering it to reappear. Can you try redoing the test? it doesn't
appear back in my tests.
                
> -Dreplace_token leaves old node (IP) in the gossip with the token.
> ------------------------------------------------------------------
>
>                 Key: CASSANDRA-3736
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3736
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.0.0
>            Reporter: Jackson Chung
>            Assignee: Vijay
>             Fix For: 1.0.8
>
>         Attachments: 0001-CASSANDRA-3736.patch
>
>
> https://issues.apache.org/jira/browse/CASSANDRA-957 introduce a -Dreplace_token,
> however, the replaced IP keeps on showing up in the Gossiper when starting the replacement
node:
> {noformat}
>  INFO [Thread-2] 2012-01-12 23:59:35,162 CassandraDaemon.java (line 213) Listening for
thrift clients...
>  INFO [GossipStage:1] 2012-01-12 23:59:35,173 Gossiper.java (line 836) Node /50.56.59.68
has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,174 Gossiper.java (line 804) InetAddress /50.56.59.68
is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,175 StorageService.java (line 988) Node /50.56.59.68
state jump to normal
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 836) Node /50.56.58.55
has restarted, now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,176 Gossiper.java (line 804) InetAddress /50.56.58.55
is now UP
>  INFO [GossipStage:1] 2012-01-12 23:59:35,177 StorageService.java (line 1016) Nodes /50.56.58.55
and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.
 Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-12 23:59:45,048 Gossiper.java (line 818) InetAddress /50.56.58.55
is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:00:06,062 Gossiper.java (line 632) FatClient /50.56.58.55
has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 838) Node /50.56.58.55
is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:01:06,320 Gossiper.java (line 804) InetAddress /50.56.58.55
is now UP
>  INFO [GossipStage:1] 2012-01-13 00:01:06,321 StorageService.java (line 1016) Nodes /50.56.58.55
and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.
 Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:01:16,106 Gossiper.java (line 818) InetAddress /50.56.58.55
is now dead.
>  INFO [GossipTasks:1] 2012-01-13 00:01:37,121 Gossiper.java (line 632) FatClient /50.56.58.55
has been silent for 30000ms, removing from gossip
>  INFO [GossipStage:1] 2012-01-13 00:02:37,352 Gossiper.java (line 838) Node /50.56.58.55
is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 Gossiper.java (line 804) InetAddress /50.56.58.55
is now UP
>  INFO [GossipStage:1] 2012-01-13 00:02:37,353 StorageService.java (line 1016) Nodes /50.56.58.55
and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.
 Ignoring /50.56.58.55
>  INFO [GossipTasks:1] 2012-01-13 00:02:47,158 Gossiper.java (line 818) InetAddress /50.56.58.55
is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,162 Gossiper.java (line 818) InetAddress /50.56.58.55
is now dead.
>  INFO [GossipStage:1] 2012-01-13 00:02:50,163 StorageService.java (line 1156) Removing
token 122029383590318827259508597176866581733 for /50.56.58.55
> {noformat}
> in the above, /50.56.58.55 was the replaced IP.
> tried adding the "Gossiper.instance.removeEndpoint(endpoint);" in the StorageService.java
where the message 'Nodes %s and %s have the same token %s.  Ignoring %s",' seems only have
fixed this temporary. Here is a ring output:
> {noformat}
> riptano@action-quick:~/work/cassandra$ ./bin/nodetool -h localhost ring
> Address         DC          Rack        Status State   Load            Owns    Token
                                      
>                                                                                85070591730234615865843651857942052864
     
> 50.56.59.68     datacenter1 rack1       Up     Normal  6.67 KB         85.56%  60502102442797279294142560823234402248
     
> 50.56.31.186    datacenter1 rack1       Up     Normal  11.12 KB        14.44%  85070591730234615865843651857942052864

> {noformat}
> gossipinfo:
> {noformat}
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.58.55
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.58.55
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> /50.56.59.68
>   LOAD:6835.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:11387.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> {noformat}
> Note that at 1 point earlier it seems to have been removed:
> $ ./bin/nodetool -h localhost gossipinfo
> /50.56.59.68
>   LOAD:13815.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.59.68
>   STATUS:NORMAL,60502102442797279294142560823234402248
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> action-quick2/50.56.31.186
>   LOAD:13725.0
>   SCHEMA:00000000-0000-1000-0000-000000000000
>   RPC_ADDRESS:50.56.31.186
>   STATUS:NORMAL,85070591730234615865843651857942052864
>   RELEASE_VERSION:1.0.7-SNAPSHOT
> riptano@action-quick2:~/work/cassandra$  INFO [GossipStage:1] 2012-01-13 01:03:30,073
Gossiper.java (line 838) Node /50.56.58.55 is now part of the cluster
>  INFO [GossipStage:1] 2012-01-13 01:03:30,073 Gossiper.java (line 804) InetAddress /50.56.58.55
is now UP
>  INFO [GossipStage:1] 2012-01-13 01:03:30,074 StorageService.java (line 1017) Nodes /50.56.58.55
and action-quick2/50.56.31.186 have the same token 85070591730234615865843651857942052864.
 Ignoring /50.56.58.55

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message