cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Ellis (Commented) (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-3533) TimeoutException when there is a firewall issue.
Date Sat, 26 Nov 2011 07:20:40 GMT


Jonathan Ellis commented on CASSANDRA-3533:

Alex Feinberg from Voldemort says:

bq. We had this situation by accident in production, when nodes in the other datacenter were
firewalled off from clients in one cluster.  The way we deal with it is, our failure detector
is local to each client and has a thread which keeps pinging each node it marked down initially
to see if it came back up. ThreadholdFailureDetector, which inherits from AsyncRecoveryFailureDetector.

> TimeoutException when there is a firewall issue.
> ------------------------------------------------
>                 Key: CASSANDRA-3533
>                 URL:
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 1.0.4
>            Reporter: Vijay
>            Priority: Minor
> When one node in the cluster is not able to talk to the other DC/RAC due to firewall
or network related issue (StorageProxy calls fail), and the nodes are NOT marked down because
at least one node in the cluster can talk to the other DC/RAC, we get timeoutException instead
of throwing a unavailableException.
> The problem with this:
> 1) It is hard to monitor/identify these errors.
> 2) It is hard to diffrentiate from the client if the node being bad vs a bad query.
> 3) when this issue happens we have to wait for at-least the RPC timeout time to know
that the query wont succeed.
> Possible Solution: when marking a node down we might want to check if the node is actually
alive by trying to communicate to it? So we can be sure that the node is actually alive.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message