incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vasileios Vlachos <vasileiosvlac...@gmail.com>
Subject Re: Node stuck during nodetool rebuild
Date Wed, 06 Aug 2014 21:08:05 GMT
Hello Mark and Rob,

Thank you very much for your input, I will increase the phi threshold and
report back any progress.

Vasilis
On 5 Aug 2014 21:52, "Mark Reddy" <mark.reddy@boxever.com> wrote:

> Hi Vasilis,
>
> To further on what Rob said
>
> I believe you might be able to tune the phi detector threshold to help
>> this operation complete, hopefully someone with direct experience of same
>> will chime in.
>
>
> I have been through this operation where streams break due to a node
> falsely being marked down (flapping). In an attempt to  mitigate this I
> increase the phi_convict_threshold in cassandra.yaml from 8 to 10, after
> which the rebuild was able to successfully complete. The default value for
> phi_convict_threshold is 8 with 12 being the maximum recommended value.
>
>
> Mark
>
>
> On Tue, Aug 5, 2014 at 7:22 PM, Robert Coli <rcoli@eventbrite.com> wrote:
>
>> On Tue, Aug 5, 2014 at 1:28 AM, Vasileios Vlachos <
>> vasileiosvlachos@gmail.com> wrote:
>>
>>> The problem is that the nodetool seems to be stuck, and nodetool
>>> netstats on node1 of DC2 appears to be stuck at 10% streaming a 5G file
>>> from node2 at DC1. This doesn't tally with nodetool netstats when running
>>> it against either of the DC1 nodes. The DC1 nodes don't think they stream
>>> anything to DC2.
>>>
>>
>> Yes, streaming is fragile and breaks and hangs forever and your only
>> option in most cases is to stop the rebuilding node, nuke its data, and
>> start again.
>>
>> I believe you might be able to tune the phi detector threshold to help
>> this operation complete, hopefully someone with direct experience of same
>> will chime in.
>>
>> =Rob
>>
>>
>
>

Mime
View raw message