incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Hadley <r...@sgizmo.com>
Subject Re: Nodetool removetoken taking days to run.
Date Wed, 14 Sep 2011 21:25:33 GMT

On Sep 14, 2011, at 2:08 PM, Brandon Williams wrote:

> On Wed, Sep 14, 2011 at 8:54 AM, Ryan Hadley <ryan@sgizmo.com> wrote:
>> Hi,
>> 
>> So, here's the backstory:
>> 
>> We were running Cassandra 0.7.4 and at one point in time had a node in the ring at
10.84.73.18. We removed this node from the ring successfully in 0.7.4. It stopped showing
in the nodetool ring command. But occasionally we'd still get weird log entries about failing
to write/read to IP 10.84.73.18.
>> 
>> We upgraded to Cassandra 0.8.4. Now, nodetool ring shows this old node:
>> 
>> 10.84.73.18     datacenter1 rack1       Down   Leaving ?               6.71%   32695837177645752437561450928649262701
>> 
>> So I started a nodetool removetoken on 32695837177645752437561450928649262701 last
Friday. It's still going strong this morning, on day 5:
>> 
>> ./bin/nodetool -h 10.84.73.47 -p 8080 removetoken status
>> RemovalStatus: Removing token (32695837177645752437561450928649262701). Waiting for
replication confirmation from [/10.84.73.49,/10.84.73.48,/10.84.73.51].
>> 
>> Should I just be patient? Or is something really weird with this node?
> 
> 5 days seems excessive unless there is a very large amount of data per
> node.  I would check nodetool netstats, and if the streams don't look
> active issue a 'removetoken force' against 10.84.73.47 and accept that
> you may possibly need to run repair to restore the replica count.
> 
> -Brandon

Hi Brandon,

Thanks for the reply. Quick question though:

1. We write all data to this ring with a TTL of 30 days
2. This node hasn't been in the ring for at least 90 days, more like 120 days since it's been
in the ring.

So, if I nodetool removetoken forced it, would I still have to be concerned about running
a repair?

Also, after this node is removed, I'm going to rebalance with nodetool move. Would that remove
the repair requirement too?

Thanks-
Ryan
Mime
View raw message