cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ron Kuris (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-9536) The failure detector becomes more sensitive when the network is flakey
Date Tue, 02 Jun 2015 19:42:50 GMT


Ron Kuris commented on CASSANDRA-9536:

There are a few possible fixes to this, each has their pros and cons:
# Don't record values less than 1 second; these are probably retransmits. This will prevent
the phi average from being dropped below 1, which means that all phi can do is make the system
less sensitive. The disadvantage to this is that flakey networks will still not receive any
phi bonuses.
# Whenever a packet is received longer than MAX_INTERVAL_IN_NANOS, just record an entry with
MAX_INTERVAL_IN_NANOS instead of just dropping it.
# Do both of the above

I am leaning toward both, since this means that flakier networks will raise the phi average
closer to 2, giving them twice the resilience before being marked down.

> The failure detector becomes more sensitive when the network is flakey
> ----------------------------------------------------------------------
>                 Key: CASSANDRA-9536
>                 URL:
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Ron Kuris
>            Priority: Minor
> I added considerable instrumentation into the failure detector, and then blocked port
7000 for a random 5-6 second interval, then resumed traffic for the same amount of time, with
a script like:
> {code}while :
> do
>    iptables -A INPUT -p tcp --destination-port 7000 -j DROP
>         v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)]
>         s=5.${v:1:2}${v:4:3}
>         echo offline for $s
>         sleep $s
>         iptables -F
>         v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)]
>         s=5.${v:1:2}${v:4:3}
>         echo online for $s
>         sleep $s
> done{code}
> When I do this, I watch the values being reported to the FailureDetector. The median
actually goes down, as low as 850ms. The reason is that the very slow packets are not recorded
(they exceed MAX_INTERVAL_IN_NANO which is 2 seconds) and the retransmitted packets arrive
very quickly in succession, lowering the overall average.
> Once the average is lowered, the node becomes much more sensitive to shorter outages.
If you run this code for a while, the average drops down to 800ms or less, which means that
the node will go down 20% quicker than expected.

This message was sent by Atlassian JIRA

View raw message