kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From benstopford <...@git.apache.org>
Subject [GitHub] kafka pull request #1893: KAFKA-4178: Windows in Rate Calculation
Date Tue, 20 Sep 2016 17:45:06 GMT
GitHub user benstopford opened a pull request:

    https://github.com/apache/kafka/pull/1893

    KAFKA-4178: Windows in Rate Calculation

    ReplicationQuotas included a new Rate class which changed the way the window is calculated.
@junrao asked that we look to consolidate this. On balance I believe there is a good case
for both, so this PR pulls out two policies for calculating the window, and attempts to explain
what they are and why we might need them. 
    
    They are:
    **Elapsed**: Set the window to the difference between the oldest and newest measurement
time. Handle NaN. _(Replication Quotas use this)_
    **Fixed**: Fix the window to the full duration (10s by default). _(Client Quotas use this)_
    
    Replication Quotas uses **Elapsed** as **Fixed** provides underestimates during the first
window. Underestimates are best avoided as they create a load spike when replication starts.
So it's preferable to overestimate the rate, and hence increase the throttled rate slowly.
Elapsed is also significantly easier to test. 
    
    I'm totally sure why Client Quotas were changed to use **Fixed**. I know there was a NaN
issue, but should really be tangential (it is handled in both policies). However it seems
sensible to slowly throttle on a client's connection down to the desired rate when you initialise
or change quotas. 
    
    So I think there is a requirement for both types of rate, and hence I've included both
in this PR. **Elapsed** suits throttled replication, where the general use case is to initiate
some rebalance and immediately apply a conservative throttle. **Fixed** suits client quotas
where we want to gently throttle a client down to the desired rate over a period of seconds.



You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/benstopford/kafka KAFKA-4178

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/kafka/pull/1893.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1893
    
----

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message