kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boyang Chen <bche...@outlook.com>
Subject Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances
Date Fri, 04 Jan 2019 21:01:08 GMT
Thanks Guozhang for the proposal! The update is done.

________________________________
From: Guozhang Wang <wangguoz@gmail.com>
Sent: Saturday, January 5, 2019 3:33 AM
To: dev
Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

Hello Boyang,

I've made another pass on the wiki page again. One minor comment is on the
"Server Behavior Changes" section, we should have a paragraph on the
logical changes on handling new versions of LeaveGroupRequest (e.g. how to
handle dynamic member v.s. static member etc).

Other than that, I do not have further comments. I think we can continue
the voting process after that.

Guozhang

On Wed, Jan 2, 2019 at 10:00 AM Boyang Chen <bchen11@outlook.com> wrote:

> Thanks Jason for the comment! I answered it on the discuss thread.
>
> Folks, could we continue the vote for this KIP? This is a very critical
> improvement for our streaming system
> stability and we need to get things rolling right at the start of 2019.
>
> Thank you for your time!
> Boyang
>
> ________________________________
> From: Jason Gustafson <jason@confluent.io>
> Sent: Tuesday, December 18, 2018 7:40 AM
> To: dev
> Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> reduce consumer rebalances
>
> Hi Boyang,
>
> Thanks, the KIP looks good. Just one comment.
>
> The new schema for the LeaveGroup request is slightly odd since it is
> handling both the single consumer use case and the administrative use case.
> I wonder we could make it consistent from a batching perspective.
>
> In other words, instead of this:
> LeaveGroupRequest => GroupId MemberId [GroupInstanceId]
>
> Maybe we could do this:
> LeaveGroupRequest => GroupId [GroupInstanceId MemberId]
>
> For dynamic members, GroupInstanceId could be empty, which is consistent
> with JoinGroup. What do you think?
>
> Also, just for clarification, what is the expected behavior if the current
> memberId of a static member is passed to LeaveGroup? Will the static member
> be removed? I know the consumer will not do this, but we'll still have to
> handle the case on the broker.
>
> Best,
> Jason
>
>
> On Mon, Dec 10, 2018 at 11:54 PM Boyang Chen <bchen11@outlook.com> wrote:
>
> > Thanks Stanislav!
> >
> > Get Outlook for iOS<https://aka.ms/o0ukef>
> >
> > ________________________________
> > From: Stanislav Kozlovski <stanislav@confluent.io>
> > Sent: Monday, December 10, 2018 11:28 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> > reduce consumer rebalances
> >
> > This is great work, Boyang. Thank you very much.
> >
> > +1 (non-binding)
> >
> > On Mon, Dec 10, 2018 at 6:09 PM Boyang Chen <bchen11@outlook.com> wrote:
> >
> > > Hey there, could I get more votes on this thread?
> > >
> > > Thanks for the vote from Mayuresh and Mike :)
> > >
> > > Best,
> > > Boyang
> > > ________________________________
> > > From: Mayuresh Gharat <gharatmayuresh15@gmail.com>
> > > Sent: Thursday, December 6, 2018 10:53 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> > > reduce consumer rebalances
> > >
> > > +1 (non-binding)
> > >
> > > Thanks,
> > >
> > > Mayuresh
> > >
> > > On Tue, Dec 4, 2018 at 6:58 AM Mike Freyberger <
> > mike.freyberger@xandr.com>
> > > wrote:
> > >
> > > > +1 (non binding)
> > > >
> > > > On 12/4/18, 9:43 AM, "Patrick Williams" <
> > patrick.williams@storageos.com
> > > >
> > > > wrote:
> > > >
> > > > Pls take me off this VOTE list
> > > >
> > > > Best,
> > > >
> > > > Patrick Williams
> > > >
> > > > Sales Manager, UK & Ireland, Nordics & Israel
> > > > StorageOS
> > > > +44 (0)7549 676279
> > > > patrick.williams@storageos.com
> > > >
> > > > 20 Midtown
> > > > 20 Proctor Street
> > > > Holborn
> > > > London WC1V 6NX
> > > >
> > > > Twitter: @patch37
> > > > LinkedIn: linkedin.com/in/patrickwilliams4 <
> > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flinkedin.com%2Fin%2Fpatrickwilliams4&amp;data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636801101252994092&amp;sdata=ipDTX%2FGARrFkwZfRuOY0M5m3iJ%2Bnkxovv6u9bBDaTyc%3D&amp;reserved=0
> > > >
> > > >
> > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslack.storageos.com%2F&amp;data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636801101252994092&amp;sdata=hxuKU6aZdQU%2FpxpqaaThR6IjpEmwIP5%2F3NhYzMYijkw%3D&amp;reserved=0
> > > >
> > > >
> > > >
> > > > On 03/12/2018, 17:34, "Guozhang Wang" <wangguoz@gmail.com> wrote:
> > > >
> > > > Hello Boyang,
> > > >
> > > > I've browsed through the new wiki and there are still a couple of
> > > > minor
> > > > things to notice:
> > > >
> > > > 1. RemoveMemberFromGroupOptions seems not defined anywhere.
> > > >
> > > > 2. LeaveGroupRequest added a list of group instance id, but still
> > > > keep the
> > > > member id as a singleton; is that intentional? I think to make
> > > the
> > > > protocol
> > > > consistent both member id and instance ids could be plural.
> > > >
> > > > 3. About the *kafka-remove-member-from-group.sh *tool, I'm
> > > > wondering if we
> > > > can defer adding this while just add the corresponding calls of
> > > the
> > > > LeaveGroupRequest inside Streams until we have used it in
> > > > production and
> > > > hence have a better understanding on how flexible or extensible
> > > if
> > > > we want
> > > > to add any cmd tools. The rationale is that if we do not
> > > > necessarily need
> > > > it now, we can always add it later with a more think-through API
> > > > design,
> > > > but if we add the tool in a rush, we may need to extend or modify
> > > > it soon
> > > > after we realize its limits in operations.
> > > >
> > > > Otherwise, I'm +1 on the proposal.
> > > >
> > > > Guozhang
> > > >
> > > >
> > > > On Mon, Dec 3, 2018 at 9:14 AM Boyang Chen <bchen11@outlook.com>
> > > > wrote:
> > > >
> > > > > Hey community friends,
> > > > >
> > > > > after another month of polishing, KIP-345<
> > > > >
> > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&amp;data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636801101252994092&amp;sdata=T4i7L1i0nIeHrrjTeLOOgYKsfzfNEMGDhTazvBEZbXw%3D&amp;reserved=0
> > > > >
> > > > > design is ready for vote. Feel free to add your comment on the
> > > > discussion
> > > > > thread or here.
> > > > >
> > > > > Thanks for your time!
> > > > >
> > > > > Boyang
> > > > > ________________________________
> > > > > From: Boyang Chen <bchen11@outlook.com>
> > > > > Sent: Friday, November 9, 2018 6:35 AM
> > > > > To: dev@kafka.apache.org
> > > > > Subject: [VOTE] KIP-345: Introduce static membership protocol
> > > to
> > > > reduce
> > > > > consumer rebalances
> > > > >
> > > > > Hey all,
> > > > >
> > > > >
> > > > > thanks so much for all the inputs on KIP-345 so far. The
> > > > original proposal
> > > > > has enhanced a lot with your help. To make sure the
> > > > implementation go
> > > > > smoothly without back and forth, I would like to start a vote
> > > on
> > > > the final
> > > > > design agreement now:
> > > > >
> > > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-&amp;data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636801101252994092&amp;sdata=g4%2BMXKpkiQLZXg5HJWfJhw1kc1PbDNwyiX9zkREVqGE%3D&amp;reserved=0
> > > <
> > > > >
> > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&amp;data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636801101252994092&amp;sdata=T4i7L1i0nIeHrrjTeLOOgYKsfzfNEMGDhTazvBEZbXw%3D&amp;reserved=0
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> 345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances<
> > > > >
> > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&amp;data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636801101252994092&amp;sdata=T4i7L1i0nIeHrrjTeLOOgYKsfzfNEMGDhTazvBEZbXw%3D&amp;reserved=0
> > > > > >
> > > > >
> > > > > KIP-345: Introduce static membership protocol to reduce ...<
> > > > >
> > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&amp;data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636801101252994092&amp;sdata=T4i7L1i0nIeHrrjTeLOOgYKsfzfNEMGDhTazvBEZbXw%3D&amp;reserved=0
> > > > > >
> > > > > cwiki.apache.org
> > > > > For stateful applications, one of the biggest performance
> > > > bottleneck is
> > > > > the state shuffling. In Kafka consumer, there is a concept
> > > called
> > > > > "rebalance" which means that for given M partitions and N
> > > > consumers in one
> > > > > consumer group, Kafka will try to balance the load between
> > > > consumers and
> > > > > ideally have ...
> > > > >
> > > > >
> > > > > Let me know if you have any questions.
> > > > >
> > > > >
> > > > > Best,
> > > > >
> > > > > Boyang
> > > > >
> > > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > > --
> > > -Regards,
> > > Mayuresh R. Gharat
> > > (862) 250-7125
> > >
> >
> >
> > --
> > Best,
> > Stanislav
> >
>


--
-- Guozhang
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message