cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From MichaƂ Michalski <>
Subject Re: Restart node => hinted handoff flood
Date Fri, 05 Jul 2013 06:33:58 GMT
My blind guess is:

In our case the only sensible solution was to pause hints delivery and 
disable storing them (both done with a nodetool: pausehandoff and 
disablehandoff). Once they TTL'd (3 hours by default I believe?) I 
turned HH on again and started to repair. However, problem has returned 
on the next day, so I had to do a quick C* upgrade with the version 
having this patch applied (we use a "self-built" 1.2.1 with a few 
additional patches applied).


W dniu 04.07.2013 18:41, Alain RODRIGUEZ pisze:
> The point is that there is no way, afaik, to limit the speed of these
> Hinted Handoff since it's not a stream like repair or bootstrap, no way
> either to keep the node out of the ring during the time it is receiving
> hints since hints and "normal" traffic both go through gossip protocol on
> port 7000.
> How to avoid this Hinted Handoff flood on returning nodes ?
> Alain
> 2013/7/4 Alain RODRIGUEZ <>
>> Hi,
>> Using C*1.2.2 12 EC2 xLarge cluster.
>> When I restart a node, if it spend a few minutes down, when I bring it up,
>> all the cpu are blocked at 100%, even once compactions are disabled,
>> inducing a very big and intolerable latency in my app. I suspect Hinted
>> Handoff to be the cause of this. disabling gossip fix the problem, enabling
>> it again brings the latency back (with a lot of gc, dropped messages...).
>> Is there a way to disable HH ? Are they responsible for this issue ?
>> I currently have this node down, any fast insight would be appreciated.
>> Alain

View raw message