Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of artur.kronenberg@openmarket.com
 designates 83.166.69.119 as permitted sender)
Message-ID: <54BD14C8.7000500@openmarket.com>
Date: Mon, 19 Jan 2015 14:29:28 +0000
From: Artur Kronenberg <artur.kronenberg@openmarket.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:31.0) Gecko/20100101 Thunderbird/31.3.0
MIME-Version: 1.0
To: user@cassandra.apache.org
Subject: Nodetool removenode stuck
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

Hi,

we have had an issue with one of our nodes today:

1. Due to a wrong setup the starting node failed to properly bootstrap. 
It was shown as UN in the cluster however did not contain any data and 
we shut it down to fix our configuration issue.

2. We figured we need to remove the node from the cluster before being 
able to restart it cleanly and have it bootstrap automatically. We used 
"nodetool removenode UUID" which caused mutliple nodes in our Datacenter 
to be marked as DOWN for some reason (taken from the log) and a bunch of 
operations against our cluster to fail. The nodes have come up again and 
other than a slight heart attack we are fine.
However, the removenode operation is now stuck and won't continue.

Can anyone recommend on how to proceed safely from here? The node is 
marked as DL in our cluster. I found 
https://issues.apache.org/jira/browse/CASSANDRA-6542 however there is no 
hint on how to handle this properly.

Is it save to use the force option here? We don't want to risk the 
cluster going down for whatever reason again.

Thank you!

Artur