cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joshua McKenzie (JIRA)" <>
Subject [jira] [Updated] (CASSANDRA-3569) Failure detector downs should not break streams
Date Tue, 13 May 2014 19:06:17 GMT


Joshua McKenzie updated CASSANDRA-3569:

    Attachment: 3569_v1.txt

Bit if a pain to test but yanking a cable and/or hard-killing a vm appears to do the trick.
 Default tcp_keepalive_intvl @ 75 and tcp_keepalive_probes @ 9 means we'll have about 16 minutes
before any StreamSession is considered dead with this v1 patch (300 seconds @ keepalive_time
+ 675 seconds for intvl * probe).

This initial version removes all Gossip and FailureDetector registration from the StreamSessions
and completely relies on tcp_keepalive_time for health detection on those components.

I'll follow up with ticket for DSE changes and open a ticket for windows install script changes
once this ticket is reviewed and any necessary changes are complete.

> Failure detector downs should not break streams
> -----------------------------------------------
>                 Key: CASSANDRA-3569
>                 URL:
>             Project: Cassandra
>          Issue Type: New Feature
>            Reporter: Peter Schuller
>            Assignee: Joshua McKenzie
>             Fix For: 2.1.1
>         Attachments: 3569-2.0.txt, 3569_v1.txt
> CASSANDRA-2433 introduced this behavior just to get repairs to don't sit there waiting
forever. In my opinion the correct fix to that problem is to use TCP keep alive. Unfortunately
the TCP keep alive period is insanely high by default on a modern Linux, so just doing that
is not entirely good either.
> But using the failure detector seems non-sensicle to me. We have a communication method
which is the TCP transport, that we know is used for long-running processes that you don't
want to incorrectly be killed for no good reason, and we are using a failure detector tuned
to detecting when not to send real-time sensitive request to nodes in order to actively kill
a working connection.
> So, rather than add complexity with protocol based ping/pongs and such, I propose that
we simply just use TCP keep alive for streaming connections and instruct operators of production
clusters to tweak net.ipv4.tcp_keepalive_{probes,intvl} as appropriate (or whatever equivalent
on their OS).
> I can submit the patch. Awaiting opinions.

This message was sent by Atlassian JIRA

View raw message