Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@kafka.apache.org
MIME-Version: 1.0
In-Reply-To: <18896B53-5722-4462-87DC-6D78B46293FB@confluent.io>
References: <FF6C1038-C69C-4698-8888-AE5531578F64@confluent.io>
 <CAAOfhrBU6vxThs23FDpg1duexuUGA8FJptn4V9k0587nSwN+hw@mail.gmail.com>
 <17A87535-3781-4625-9D03-CA0096E5F884@confluent.io> <CAFc58G-5EWcVoFhhU6CwycvGhgRMRwOv_U0oifr44wKfpzhhSg@mail.gmail.com>
 <CAAOfhrAKjs2ashiUgm2hu224hwUPfkwfSmSp+te2SkPsaSGQ2g@mail.gmail.com>
 <CAJS3ho-4duNwiaD62MNZ6BLUfftCxMXQ=EJvnW8uGT73vNXLYw@mail.gmail.com>
 <CAFc58G8dEcH+Ey38KA8+2fNaJt2572GzPohoWzJQbZeZ7Ov2sQ@mail.gmail.com>
 <CAAOfhrBKZXvgX6R_9E1v3QVHJB-r9z3sYS3F02AeBdJTXc3+JQ@mail.gmail.com>
 <CAFc58G8rsZkJcnwV+R1q1zTu1sadMRcMkt8==E=5FwMg36xACw@mail.gmail.com>
 <F22C59D2-BDCA-4C5F-9341-CE803CC2A93C@confluent.io> <CAJS3ho9HjdoZcg5cOimWfSizMb_FuB0wmnT_Uf_oSXwUvMxpjw@mail.gmail.com>
 <CAFc58G9Uf6S4vjj3bF0fuNN342egurK6t=m=fjQhs7DAeqk-6g@mail.gmail.com>
 <CAAOfhrAXyqaZYHUg==aeuACKX21QtWPo_ny2aGsgvOWcPQp8fQ@mail.gmail.com>
 <CAFc58G-J8E_5C56V0sFx3rhChXve5RaL5RqvZCoHDnKhOA-Y-A@mail.gmail.com>
 <CAJS3ho_UQ324D0=8UF3xmGX=etAQG1B1PNffxfPoFBxw+AXndA@mail.gmail.com>
 <CAFc58G-gxRQ7dD6iVstONMUfNZ+ucyhdz_5=htiTKh9thNMzcw@mail.gmail.com>
 <CALS7w+v2Hft7Zv7NMVmriQL=aY83XXwwg8j+Y=iiovN_VzJSyw@mail.gmail.com>
 <CAAOfhrCxG-A6h0TqqVDmC9+1ESH9971tPB_0bLnVTSgk7Xyrcg@mail.gmail.com>
 <5525CE9E-741B-42A2-BE84-B5A3B3007A8B@confluent.io> <CAFc58G-VXks=fk-8WdjyySJB9ObCO+mKF4pVb5+9rnXgdHCRvg@mail.gmail.com>
 <CAAOfhrB4o2_0EscHzBgKS38_jgpsu0SjUHuFtBOKuMomrWZo7w@mail.gmail.com>
 <CAFc58G-CNsH72p4LEKHsWtEQmxL7FXkZX2g_oBhXUGaDzRgx=w@mail.gmail.com>
 <CAA9QSJJgpuuQUCGuo8tCtDv50MAXCMAhunM50ymR0y1_q7WuiQ@mail.gmail.com>
 <CACrdVJo7XwqNpQjEmQKSR0rO4GHYGjOkbaMms4xi0czO6txsGg@mail.gmail.com>
 <CAA9QSJJm7eqw=eAGUTfyuzwB18q_G2cNSEoLsCLb0+frC5-Peg@mail.gmail.com> <18896B53-5722-4462-87DC-6D78B46293FB@confluent.io>
From: Jun Rao <jun@confluent.io>
Date: Mon, 26 Sep 2016 09:38:54 -0700
Message-ID: <CAFc58G_yP5eYtFcKZhqYLpwibe3Jy1GC04GK8riaCKRTycz+jA@mail.gmail.com>
Subject: Re: [DISCUSS] KIP-73: Replication Quotas
To: "dev@kafka.apache.org" <dev@kafka.apache.org>
Content-Type: multipart/alternative; boundary=001a113ed2d0989d0b053d6bc355
archived-at: Mon, 26 Sep 2016 16:39:06 -0000

--001a113ed2d0989d0b053d6bc355
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Yes, this change makes sense to me since it gives the admin better control.

Thanks,

Jun

On Sun, Sep 25, 2016 at 7:35 AM, Ben Stopford <ben@confluent.io> wrote:

> Hi All
>
> We=E2=80=99ve made an adjustment to KIP-73: Replication Quotas which I=E2=
=80=99d like to
> open up to the community for approval.
>
> Previously the admin passed a list of replicas to be throttled:
>
>         quota.replication.throttled.replicas =3D
> [partId]:[replica],[partId]:[replica],[partId]:[replica] etc
>
> The change is to split this into two properties. One that corresponds to
> the leader-side throttle, and the other that corresponds to the
> follower-side throttle.
>
>         quota.leader.replication.throttled.replicas =3D
> [partId]:[replica],[partId]:[replica],[partId]:[replica]
>         quota.follower.replication.throttled.replicas =3D
> [partId]:[replica],[partId]:[replica],[partId]:[replica]
>
> This provides more control over the throttling process. It also helps us
> with a common use case which I=E2=80=99ve described below, for those inte=
rested.
>
> Please reply if you have any comments / issues / suggestions.
>
> Thanks as ever.
>
> Ben
>
>
> Sample Use Case:
>
> Say we are moving partition 0. It has replicas [104,107,109] moving to
> [105,107,113]
>
> So the leader could be any of [104,107,109] and we know we will be
> creating new replicas on 105 & 113.
>
> In the current mechanism, all we can do is apply both leader and follower
> throttles to all 5:  [104,107,109,105,113] which will mean the regular
> replication traffic between (say 107 is leader) 107->104 and 107->109 wil=
l
> be throttled also. This is potentially problematic.
>
> What we really want to do is apply:
>
> LeaderThrottle [104,107,109]
> FollowerThrottle [105,113]
>
> This way, during a rebalance, we that standard replication traffic will
> not be  throttled, but the rebalance will perform correctly if leaders
> move. One subtlety is that, should the leader move to the =E2=80=9Cmove
> destination=E2=80=9D node, it would no longer be throttled. But this is a=
ctually to
> our benefit in the normal case.
>
>

--001a113ed2d0989d0b053d6bc355--