incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Reddy <mark.re...@boxever.com>
Subject Re: Node stuck during nodetool rebuild
Date Tue, 05 Aug 2014 18:52:27 GMT
Hi Vasilis,

To further on what Rob said

I believe you might be able to tune the phi detector threshold to help this
> operation complete, hopefully someone with direct experience of same will
> chime in.


I have been through this operation where streams break due to a node
falsely being marked down (flapping). In an attempt to  mitigate this I
increase the phi_convict_threshold in cassandra.yaml from 8 to 10, after
which the rebuild was able to successfully complete. The default value for
phi_convict_threshold is 8 with 12 being the maximum recommended value.


Mark


On Tue, Aug 5, 2014 at 7:22 PM, Robert Coli <rcoli@eventbrite.com> wrote:

> On Tue, Aug 5, 2014 at 1:28 AM, Vasileios Vlachos <
> vasileiosvlachos@gmail.com> wrote:
>
>> The problem is that the nodetool seems to be stuck, and nodetool netstats
>> on node1 of DC2 appears to be stuck at 10% streaming a 5G file from node2
>> at DC1. This doesn't tally with nodetool netstats when running it against
>> either of the DC1 nodes. The DC1 nodes don't think they stream anything to
>> DC2.
>>
>
> Yes, streaming is fragile and breaks and hangs forever and your only
> option in most cases is to stop the rebuilding node, nuke its data, and
> start again.
>
> I believe you might be able to tune the phi detector threshold to help
> this operation complete, hopefully someone with direct experience of same
> will chime in.
>
> =Rob
>
>

Mime
View raw message