Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Thu, 22 Sep 2011 19:40:27 +0000 (UTC)
From: "Jason Harvey (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: 
 <1298374632.3428.1316720427974.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <1489242129.1219.1316676086193.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (CASSANDRA-3243) Node which was decommissioned
 and shut-down reappears on a single node
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CASSANDRA-3243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13112852#comment-13112852 ] 

Jason Harvey commented on CASSANDRA-3243:
-----------------------------------------

bq. Can you explain what you mean by "dead gossip list" and how this prevents truncate?

The decommissioned node is showing up in the 'UNREACHABLE' list when calling 'describe cluster'. When I attempt to run truncate, the command returns that truncate cannot occur due to a node being down.

bq. After CASSANDRA-2496, we store dead gossip states for 3 days, so that any other nodes that were down at the time of removal can know later not to repopulate the ring with the removed node, but this isn't persisted anywhere, so since you did a full ring restart, the only candidate left is the persisted endpoints, though all nodes should have removed it from there after the decommission/removetoken.

Is there a way I can get a list of endpoints to see how this node showed back up?


Also, any thoughts on why this node only re-appeared on a single node?

Thanks!
Jason

> Node which was decommissioned and shut-down reappears on a single node
> ----------------------------------------------------------------------
>
>                 Key: CASSANDRA-3243
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3243
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 0.8.5
>            Reporter: Jason Harvey
>            Assignee: Brandon Williams
>            Priority: Minor
>
> I decommissioned a node several days ago. It was no longer in the ring list on any node in the ring. However, it was in the dead gossip list.
> In an attempt to clean it out of the dead gossip list so I could truncate, I shut down the entire ring and bought it back up. Once the ring came back up, one node showed the decommissioned node as still in the ring in a state of 'Down'. No other node in the ring shows this info.
> I successfully ran removetoken on the node to get that phantom node out. However, it is back in the dead gossip list, preventing me from truncating.
> Where might the info on this decommissioned node be being stored? Is HH possibly trying to deliver to the removed node, thus putting it back in the ring on one node?
> I find it extremely curious that none of the other nodes in the ring showed the phantom node. Shouldn't gossip have propagated the node everywhere, even if it was down?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira