cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brandon Williams (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-11892) Can not replace a dead host
Date Mon, 27 Jun 2016 19:42:52 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-11892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351670#comment-15351670
] 

Brandon Williams commented on CASSANDRA-11892:
----------------------------------------------

Your real problem seems to be a missing host_id for the down node.  You'll probably need to
assassinate it.

> Can not replace a dead host
> ---------------------------
>
>                 Key: CASSANDRA-11892
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11892
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Dikang Gu
>         Attachments: 0001-handle-hibernate-case.patch
>
>
> I got some errors when trying to replace a dead host.
> {code}
> 2016-05-25_20:59:37.61838 ERROR 20:59:37 [main]: Exception encountered during startup
> 2016-05-25_20:59:37.61839 java.lang.UnsupportedOperationException: Cannot replace token
100284002935427428580945058996711341062 which does not exist!
> 2016-05-25_20:59:37.61839       at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:925)
~[apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
> 2016-05-25_20:59:37.61839       at org.apache.cassandra.service.StorageService.initServer(StorageService.java:740)
~[apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
> 2016-05-25_20:59:37.61839       at org.apache.cassandra.service.StorageService.initServer(StorageService.java:617)
~[apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
> 2016-05-25_20:59:37.61840       at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:389)
[apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
> 2016-05-25_20:59:37.61840       at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:564)
[apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
> 2016-05-25_20:59:37.61841       at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:653)
[apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
> 2016-05-25_20:59:37.61910 Exception encountered during startup: Cannot replace token
100284002935427428580945058996711341062 which does not exist!
> {code}
> the status of the node is DN:
> {code}
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address                          Load       Tokens  Owns    Host ID             
                 Rack
> DN  2401:db00:2050:4196:face:0:13:0  809.83 GB  256     ?       null                
                 ash5-04-pp
> {code}
> I add some logging and find something like this:
> {code}
> 2016-05-25_20:58:33.44305 INFO  20:58:33 [main]: Gathering node replacement information
for /2401:db00:2050:4196:face:0:13:0
> 2016-05-25_20:58:34.36966 INFO  20:58:34 [GossipStage:1]: InetAddress /2401:db00:2050:4196:face:0:13:0
is now DOWN
> 2016-05-25_20:58:41.12167 INFO  20:58:41 [GossipStage:1]: InetAddress /2401:db00:2050:4196:face:0:13:0
is now DOWN
> 2016-05-25_20:58:41.12248 INFO  20:58:41 [GossipStage:1]: Node /2401:db00:2050:4196:face:0:13:0
state STATUS
> 2016-05-25_20:58:41.12250 INFO  20:58:41 [GossipStage:1]: Node /2401:db00:2050:4196:face:0:13:0
movename hibernate
> 2016-05-25_20:58:41.12252 INFO  20:58:41 [GossipStage:1]: Node /2401:db00:2050:4196:face:0:13:0
state LOAD
> {code}
> I find in the StorageService.onChange, we do not handle the "hibernate" VersionValue,
does it cause the problem?
> Is it safe to apply the patch to fix it?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message