Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Mon, 2 Nov 2015 00:57:27 +0000 (UTC)
From: "Stefania (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.12862975.1441868326000.120336.1446425847851@Atlassian.JIRA>
In-Reply-To: <JIRA.12862975.1441868326000@Atlassian.JIRA>
References: <JIRA.12862975.1441868326000@Atlassian.JIRA>
 <JIRA.12862975.1441868326749@arcas>
Subject: [jira] [Commented] (CASSANDRA-10298) Replaced dead node stayed in
 gossip forever
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CASSANDRA-10298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14984617#comment-14984617 ] 

Stefania commented on CASSANDRA-10298:
--------------------------------------

+1, pending CI here:

http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-10298-2.1-testall/
http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-10298-2.1-dtest/

The code that catches the exception in {{handleStateLeft()}} will be improved by CASSANDRA-10089 where we will return an empty collection in case of no tokens therefore removing the need for a try/catch. We must also remember to check for {{!tokens.isEmpty()}} in {{excise()}} but we'll deal with this in 10089.

A test to reproduce this problem would be helpful to ensure we don't break it again later on. [~dikanggu] are you able to reproduce this regularly and if so what are the exact steps?


> Replaced dead node stayed in gossip forever
> -------------------------------------------
>
>                 Key: CASSANDRA-10298
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10298
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Dikang Gu
>            Assignee: Dikang Gu
>             Fix For: 2.1.x
>
>         Attachments: CASSANDRA-10298.patch
>
>
> The dead node stayed in the nodetool status,
> DN  10.210.165.55                    379.76 GB  256     ?       null
> And in the log, it throws NPE when trying to remove it.
> {code}
> 2015-09-10_06:41:22.92453 ERROR 06:41:22 Exception in thread Thread[GossipStage:1,5,main]
> 2015-09-10_06:41:22.92454 java.lang.NullPointerException: null
> 2015-09-10_06:41:22.92455       at org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:100) 
> 2015-09-10_06:41:22.92455       at org.apache.cassandra.db.HintedHandOffManager.deleteHintsForEndpoint(HintedHandOffManager.java:201) 
> 2015-09-10_06:41:22.92455       at org.apache.cassandra.service.StorageService.excise(StorageService.java:1886) 
> 2015-09-10_06:41:22.92455       at org.apache.cassandra.service.StorageService.excise(StorageService.java:1902) 
> 2015-09-10_06:41:22.92456       at org.apache.cassandra.service.StorageService.handleStateLeft(StorageService.java:1805)
> 2015-09-10_06:41:22.92457       at org.apache.cassandra.service.StorageService.onChange(StorageService.java:1473) 
> 2015-09-10_06:41:22.92457       at org.apache.cassandra.service.StorageService.onJoin(StorageService.java:2099) 
> 2015-09-10_06:41:22.92457       at org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:1009) 
> 2015-09-10_06:41:22.92458       at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1085) 
> 2015-09-10_06:41:22.92458       at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:49) 
> 2015-09-10_06:41:22.92458       at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) 
> 2015-09-10_06:41:22.92459       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_45]
> 2015-09-10_06:41:22.92460       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_45]
> {code}


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)