Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 3A93D200B26 for ; Mon, 27 Jun 2016 21:42:54 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 362AE160A54; Mon, 27 Jun 2016 19:42:54 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 82064160A62 for ; Mon, 27 Jun 2016 21:42:53 +0200 (CEST) Received: (qmail 51012 invoked by uid 500); 27 Jun 2016 19:42:52 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 50887 invoked by uid 99); 27 Jun 2016 19:42:52 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2016 19:42:52 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 330FB2C1F60 for ; Mon, 27 Jun 2016 19:42:52 +0000 (UTC) Date: Mon, 27 Jun 2016 19:42:52 +0000 (UTC) From: "Brandon Williams (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CASSANDRA-11892) Can not replace a dead host MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Mon, 27 Jun 2016 19:42:54 -0000 [ https://issues.apache.org/jira/browse/CASSANDRA-11892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351670#comment-15351670 ] Brandon Williams commented on CASSANDRA-11892: ---------------------------------------------- Your real problem seems to be a missing host_id for the down node. You'll probably need to assassinate it. > Can not replace a dead host > --------------------------- > > Key: CASSANDRA-11892 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11892 > Project: Cassandra > Issue Type: Bug > Reporter: Dikang Gu > Attachments: 0001-handle-hibernate-case.patch > > > I got some errors when trying to replace a dead host. > {code} > 2016-05-25_20:59:37.61838 ERROR 20:59:37 [main]: Exception encountered during startup > 2016-05-25_20:59:37.61839 java.lang.UnsupportedOperationException: Cannot replace token 100284002935427428580945058996711341062 which does not exist! > 2016-05-25_20:59:37.61839 at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:925) ~[apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267] > 2016-05-25_20:59:37.61839 at org.apache.cassandra.service.StorageService.initServer(StorageService.java:740) ~[apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267] > 2016-05-25_20:59:37.61839 at org.apache.cassandra.service.StorageService.initServer(StorageService.java:617) ~[apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267] > 2016-05-25_20:59:37.61840 at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:389) [apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267] > 2016-05-25_20:59:37.61840 at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:564) [apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267] > 2016-05-25_20:59:37.61841 at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:653) [apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267] > 2016-05-25_20:59:37.61910 Exception encountered during startup: Cannot replace token 100284002935427428580945058996711341062 which does not exist! > {code} > the status of the node is DN: > {code} > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- Address Load Tokens Owns Host ID Rack > DN 2401:db00:2050:4196:face:0:13:0 809.83 GB 256 ? null ash5-04-pp > {code} > I add some logging and find something like this: > {code} > 2016-05-25_20:58:33.44305 INFO 20:58:33 [main]: Gathering node replacement information for /2401:db00:2050:4196:face:0:13:0 > 2016-05-25_20:58:34.36966 INFO 20:58:34 [GossipStage:1]: InetAddress /2401:db00:2050:4196:face:0:13:0 is now DOWN > 2016-05-25_20:58:41.12167 INFO 20:58:41 [GossipStage:1]: InetAddress /2401:db00:2050:4196:face:0:13:0 is now DOWN > 2016-05-25_20:58:41.12248 INFO 20:58:41 [GossipStage:1]: Node /2401:db00:2050:4196:face:0:13:0 state STATUS > 2016-05-25_20:58:41.12250 INFO 20:58:41 [GossipStage:1]: Node /2401:db00:2050:4196:face:0:13:0 movename hibernate > 2016-05-25_20:58:41.12252 INFO 20:58:41 [GossipStage:1]: Node /2401:db00:2050:4196:face:0:13:0 state LOAD > {code} > I find in the StorageService.onChange, we do not handle the "hibernate" VersionValue, does it cause the problem? > Is it safe to apply the patch to fix it? -- This message was sent by Atlassian JIRA (v6.3.4#6332)