cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brandon Williams (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-5272) Hinted Handoff Throttle based on cluster size
Date Tue, 28 May 2013 20:01:20 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-5272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668609#comment-13668609
] 

Brandon Williams commented on CASSANDRA-5272:
---------------------------------------------

Would it be better perhaps to throttle by the amount of live nodes?  It won't always be accurate,
but if you lost a WAN link between DCs you'd be throttling a lot more than needed at the time.
                
> Hinted Handoff Throttle based on cluster size
> ---------------------------------------------
>
>                 Key: CASSANDRA-5272
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5272
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 1.2.0
>            Reporter: Rick Branson
>            Assignee: Jonathan Ellis
>            Priority: Minor
>              Labels: lhf
>             Fix For: 1.2.6
>
>         Attachments: 5272.txt
>
>
> For a 12-node EC2 m1.xlarge cluster, restarting a node causes it to get completely overloaded
with the default 2-thread, 1024KB setting in 1.2.x. This seemed to be a smaller problem when
it was 6-nodes, but still required us to abort handoffs. The old defaults in 1.1.x were WAY
more conservative. I've dropped this way down to 128KB on our production cluster which is
really conservative, but appears to have solved it. The default seems way too high on any
cluster that is non-trivial in size.
> After putting some thought to this, it seems that this should really be based on cluster
size, making the throttle a "target" for how much write load a single node can swallow. As
the cluster grows, the amount of hints that can be delivered by each other node in the cluster
goes down, so the throttle should self-adjust to take that into account.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message