incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brandon Williams <dri...@gmail.com>
Subject Re: Nodetool removetoken taking days to run.
Date Wed, 14 Sep 2011 20:08:52 GMT
On Wed, Sep 14, 2011 at 8:54 AM, Ryan Hadley <ryan@sgizmo.com> wrote:
> Hi,
>
> So, here's the backstory:
>
> We were running Cassandra 0.7.4 and at one point in time had a node in the ring at 10.84.73.18.
We removed this node from the ring successfully in 0.7.4. It stopped showing in the nodetool
ring command. But occasionally we'd still get weird log entries about failing to write/read
to IP 10.84.73.18.
>
> We upgraded to Cassandra 0.8.4. Now, nodetool ring shows this old node:
>
> 10.84.73.18     datacenter1 rack1       Down   Leaving ?               6.71%
  32695837177645752437561450928649262701
>
> So I started a nodetool removetoken on 32695837177645752437561450928649262701 last Friday.
It's still going strong this morning, on day 5:
>
> ./bin/nodetool -h 10.84.73.47 -p 8080 removetoken status
> RemovalStatus: Removing token (32695837177645752437561450928649262701). Waiting for replication
confirmation from [/10.84.73.49,/10.84.73.48,/10.84.73.51].
>
> Should I just be patient? Or is something really weird with this node?

5 days seems excessive unless there is a very large amount of data per
node.  I would check nodetool netstats, and if the streams don't look
active issue a 'removetoken force' against 10.84.73.47 and accept that
you may possibly need to run repair to restore the replica count.

-Brandon

Mime
View raw message