kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <...@confluent.io>
Subject Re: [DISCUSS] KIP-73: Replication Quotas
Date Mon, 26 Sep 2016 16:38:54 GMT
Yes, this change makes sense to me since it gives the admin better control.

Thanks,

Jun

On Sun, Sep 25, 2016 at 7:35 AM, Ben Stopford <ben@confluent.io> wrote:

> Hi All
>
> We’ve made an adjustment to KIP-73: Replication Quotas which I’d like to
> open up to the community for approval.
>
> Previously the admin passed a list of replicas to be throttled:
>
>         quota.replication.throttled.replicas =
> [partId]:[replica],[partId]:[replica],[partId]:[replica] etc
>
> The change is to split this into two properties. One that corresponds to
> the leader-side throttle, and the other that corresponds to the
> follower-side throttle.
>
>         quota.leader.replication.throttled.replicas =
> [partId]:[replica],[partId]:[replica],[partId]:[replica]
>         quota.follower.replication.throttled.replicas =
> [partId]:[replica],[partId]:[replica],[partId]:[replica]
>
> This provides more control over the throttling process. It also helps us
> with a common use case which I’ve described below, for those interested.
>
> Please reply if you have any comments / issues / suggestions.
>
> Thanks as ever.
>
> Ben
>
>
> Sample Use Case:
>
> Say we are moving partition 0. It has replicas [104,107,109] moving to
> [105,107,113]
>
> So the leader could be any of [104,107,109] and we know we will be
> creating new replicas on 105 & 113.
>
> In the current mechanism, all we can do is apply both leader and follower
> throttles to all 5:  [104,107,109,105,113] which will mean the regular
> replication traffic between (say 107 is leader) 107->104 and 107->109 will
> be throttled also. This is potentially problematic.
>
> What we really want to do is apply:
>
> LeaderThrottle [104,107,109]
> FollowerThrottle [105,113]
>
> This way, during a rebalance, we that standard replication traffic will
> not be  throttled, but the rebalance will perform correctly if leaders
> move. One subtlety is that, should the leader move to the “move
> destination” node, it would no longer be throttled. But this is actually to
> our benefit in the normal case.
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message