cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Ellis (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-2034) Make Read Repair unnecessary when Hinted Handoff is enabled
Date Thu, 12 May 2011 21:39:47 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032696#comment-13032696
] 

Jonathan Ellis commented on CASSANDRA-2034:
-------------------------------------------

bq. there is the potential to take us back to the Bad Old Days when HH could cause cascading
failure

To elaborate, the scenario here is, we did a write that succeeded on some nodes, but not others.
So we need to write a local hint to replay to the down-or-slow nodes later. But, those nodes
being down-or-slow mean load has increased on the rest of the cluster, and writing the extra
hint will increase that further, possibly enough that other nodes will see this coordinator
as down-or-slow, too, and so on.

So I think what we want to do, with this option on, is to attempt the hint write but if we
can't do it in a reasonable time, throw back a TimedOutException which is already our signal
that "your cluster may be overloaded, you need to back off."

Specifically, we could add a separate executor here, with a blocking, capped queue. When we
go to do a hint-after-failure we enqueue the write but if it is rejected because queue is
full we throw the TOE. Otherwise, we wait for the write and then return success to the client.

The tricky part is the queue needs to be large enough to handle load spikes but small enough
that wait-for-success-post-enqueue is negligible compared to RpcTimeout. If we had different
timeouts for writes than reads (which we don't -- CASSANDRA-959) then it might be nice to
use say 80% of the timeout for the normal write, and reserve 20% for the hint phase.

> Make Read Repair unnecessary when Hinted Handoff is enabled
> -----------------------------------------------------------
>
>                 Key: CASSANDRA-2034
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2034
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Jonathan Ellis
>            Assignee: Jonathan Ellis
>             Fix For: 1.0
>
>   Original Estimate: 8h
>  Remaining Estimate: 8h
>
> Currently, HH is purely an optimization -- if a machine goes down, enabling HH means
RR/AES will have less work to do, but you can't disable RR entirely in most situations since
HH doesn't kick in until the FailureDetector does.
> Let's add a scheduled task to the mutate path, such that we return to the client normally
after ConsistencyLevel is achieved, but after RpcTimeout we check the responseHandler write
acks and write local hints for any missing targets.
> This would making disabling RR when HH is enabled a much more reasonable option, which
has a huge impact on read throughput.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message