Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 23617187D3 for ; Mon, 14 Sep 2015 20:50:08 +0000 (UTC) Received: (qmail 76990 invoked by uid 500); 14 Sep 2015 20:49:46 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 76955 invoked by uid 500); 14 Sep 2015 20:49:46 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 76939 invoked by uid 99); 14 Sep 2015 20:49:45 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Sep 2015 20:49:45 +0000 Date: Mon, 14 Sep 2015 20:49:45 +0000 (UTC) From: "Jim Witschey (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CASSANDRA-10298) Replaced dead node stayed in gossip forever MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-10298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744236#comment-14744236 ] Jim Witschey commented on CASSANDRA-10298: ------------------------------------------ [~dikanggu] Is this related to CASSANDRA-10321, and if so, how? You mentioned [in this comment|https://issues.apache.org/jira/browse/CASSANDRA-10321?focusedCommentId=14744120&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14744120] that the node was running, but marked as having Thrift and Gossip unavailable. Is that the same node you tried to remove? and could that unavailable state be the result of the failed removal? > Replaced dead node stayed in gossip forever > ------------------------------------------- > > Key: CASSANDRA-10298 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10298 > Project: Cassandra > Issue Type: Bug > Reporter: Dikang Gu > Attachments: CASSANDRA-10298.patch > > > The dead node stayed in the nodetool status, > DN 10.210.165.55 379.76 GB 256 ? null > And in the log, it throws NPE when trying to remove it. > 2015-09-10_06:41:22.92453 ERROR 06:41:22 Exception in thread Thread[GossipStage:1,5,main] > 2015-09-10_06:41:22.92454 java.lang.NullPointerException: null > 2015-09-10_06:41:22.92455 at org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:100) > 2015-09-10_06:41:22.92455 at org.apache.cassandra.db.HintedHandOffManager.deleteHintsForEndpoint(HintedHandOffManager.java:201) > 2015-09-10_06:41:22.92455 at org.apache.cassandra.service.StorageService.excise(StorageService.java:1886) > 2015-09-10_06:41:22.92455 at org.apache.cassandra.service.StorageService.excise(StorageService.java:1902) > 2015-09-10_06:41:22.92456 at org.apache.cassandra.service.StorageService.handleStateLeft(StorageService.java:1805) > 2015-09-10_06:41:22.92457 at org.apache.cassandra.service.StorageService.onChange(StorageService.java:1473) > 2015-09-10_06:41:22.92457 at org.apache.cassandra.service.StorageService.onJoin(StorageService.java:2099) > 2015-09-10_06:41:22.92457 at org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:1009) > 2015-09-10_06:41:22.92458 at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1085) > 2015-09-10_06:41:22.92458 at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:49) > 2015-09-10_06:41:22.92458 at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) > 2015-09-10_06:41:22.92459 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_45] > 2015-09-10_06:41:22.92460 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_45] -- This message was sent by Atlassian JIRA (v6.3.4#6332)