incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ike Walker <ike.wal...@flite.com>
Subject Re: Long running nodetool move operation
Date Wed, 11 Sep 2013 13:07:44 GMT
The restart worked.

Thanks, Rob!

After the restart I ran 'nodetool move' again, used 'nodetool netstats | grep -v "0%"' to
verify that data was actively streaming, and the move completed successfully.

-Ike

On Sep 10, 2013, at 11:04 AM, Ike Walker <ike.walker@flite.com> wrote:

> Below is the output of "nodetool netstats".
> 
> I've never run that before, but from what I can read it shows no incoming streams, and
a bunch of outgoing streams to two other nodes, all at 0%.
> 
> I'll try the restart.
> 
> Thanks.
> 
> nodetool netstats
> Mode: MOVING
> Streaming to: /10.xxx.xx.xx
> <lots of files, all 0%>
> ...
> Streaming to: /10.xxx.xx.xxx
> <lots of files, all 0%>
> ...
> Not receiving any streams.
> Pool Name                    Active   Pending      Completed
> Commands                        n/a         0      243401039
> Responses                       n/a         0      295522535
> 
> On Sep 9, 2013, at 10:54 PM, Robert Coli <rcoli@eventbrite.com> wrote:
> 
>>   On Mon, Sep 9, 2013 at 7:08 PM, Ike Walker <ike.walker@flite.com> wrote:
>> I've been using nodetool move to rebalance my cluster. Most of the moves take under
an hour, or a few hours at most. The current move has taken 4+ days so I'm afraid it will
never complete. What's the best way to cancel it and try again?
>> 
>> What does "nodetool netstats" say? If it shows no streams in progress, the move is
probably hung...
>> 
>> Restart the affected node. If that doesn't work, restart other nodes which might
have been receiving a stream. I think in the case of "move" it should work to just restart
the affected node. Restart the move, you will re-stream anything you already streamed once.
>> 
>> https://issues.apache.org/jira/browse/CASSANDRA-3486
>> 
>> If this ticket were completed, it would presumably include the ability to stop other
hung streaming operations, like "move".
>> 
>> =Rob
> 


Mime
View raw message