incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Node failing to decomission (vnodes and 1.2.5)
Date Tue, 18 Jun 2013 08:10:22 GMT
> I also am not seeing anything in the nodes log files to suggest errors during streaming
or leaving.
You should see a log message saying "DECOMMISSIONED" when the process completes. 

What does nodetool status say?

> What suggestions does anyone have on getting this node removed from my ring so I can
rebuild it with the correct number of tokens, before I end up with a disk space issue from
too many vnodes.
If you really want to get the node out of there shut it down and run nodetool removenode on
one of the remaining nodes. 

Cheers
 

-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 18/06/2013, at 2:59 PM, David McNelis <dmcnelis@gmail.com> wrote:

> I have a node in my ring (1.2.5) that when it was set up, had the wrong number of vnodes
assigned (double the amount it should have had).
> 
> As  a result, and because we can't reduce the number of vnodes on a machine (at least
at this point), I need to decommission the node.
> 
> The problem is that we've tried running decommission several times.  In each instance
we'll have a lot of streams to other nodes for a period, and then eventually, netstats will
tell us:
> 
> nodetool -h localhost netstats
> Mode: LEAVING
>  Nothing streaming to /10.x.x.1
>  Nothing streaming to /10.x.x.2
>  Nothing streaming to /10.x.x.3
> Not receiving any streams.
> Pool Name                    Active   Pending      Completed
> Commands                        n/a         0         955991
> Responses                       n/a         0        2947860
> 
> I also am not seeing anything in the nodes log files to suggest errors during streaming
or leaving.
> 
> Then the node will stay in this leaving state for... well, we gave up after several days
of no more activity and retried several times.  Each time we "gave up" on it, we restarted
the service and it was no longer listed as Leaving, just active.  Even when in a "leaving"
state, the size of data on the node continued to grow.
> 
> What suggestions does anyone have on getting this node removed from my ring so I can
rebuild it with the correct number of tokens, before I end up with a disk space issue from
too many vnodes.


Mime
View raw message