kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-4178) Replication Throttling: Consolidate Rate Classes
Date Tue, 20 Sep 2016 17:45:22 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15507247#comment-15507247

ASF GitHub Bot commented on KAFKA-4178:

GitHub user benstopford opened a pull request:


    KAFKA-4178: Windows in Rate Calculation

    ReplicationQuotas included a new Rate class which changed the way the window is calculated.
@junrao asked that we look to consolidate this. On balance I believe there is a good case
for both, so this PR pulls out two policies for calculating the window, and attempts to explain
what they are and why we might need them. 
    They are:
    **Elapsed**: Set the window to the difference between the oldest and newest measurement
time. Handle NaN. _(Replication Quotas use this)_
    **Fixed**: Fix the window to the full duration (10s by default). _(Client Quotas use this)_
    Replication Quotas uses **Elapsed** as **Fixed** provides underestimates during the first
window. Underestimates are best avoided as they create a load spike when replication starts.
So it's preferable to overestimate the rate, and hence increase the throttled rate slowly.
Elapsed is also significantly easier to test. 
    I'm totally sure why Client Quotas were changed to use **Fixed**. I know there was a NaN
issue, but should really be tangential (it is handled in both policies). However it seems
sensible to slowly throttle on a client's connection down to the desired rate when you initialise
or change quotas. 
    So I think there is a requirement for both types of rate, and hence I've included both
in this PR. **Elapsed** suits throttled replication, where the general use case is to initiate
some rebalance and immediately apply a conservative throttle. **Fixed** suits client quotas
where we want to gently throttle a client down to the desired rate over a period of seconds.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/benstopford/kafka KAFKA-4178

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1893


> Replication Throttling: Consolidate Rate Classes
> ------------------------------------------------
>                 Key: KAFKA-4178
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4178
>             Project: Kafka
>          Issue Type: Improvement
>          Components: replication
>    Affects Versions:
>            Reporter: Ben Stopford
> Replication throttling is using a different implementation of Rate to client throttling
(Rate & SimpleRate). These should be consolidated so both use the same approach. 

This message was sent by Atlassian JIRA

View raw message