kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances
Date Fri, 04 Jan 2019 19:33:37 GMT
Hello Boyang,

I've made another pass on the wiki page again. One minor comment is on the
"Server Behavior Changes" section, we should have a paragraph on the
logical changes on handling new versions of LeaveGroupRequest (e.g. how to
handle dynamic member v.s. static member etc).

Other than that, I do not have further comments. I think we can continue
the voting process after that.

Guozhang

On Wed, Jan 2, 2019 at 10:00 AM Boyang Chen <bchen11@outlook.com> wrote:

> Thanks Jason for the comment! I answered it on the discuss thread.
>
> Folks, could we continue the vote for this KIP? This is a very critical
> improvement for our streaming system
> stability and we need to get things rolling right at the start of 2019.
>
> Thank you for your time!
> Boyang
>
> ________________________________
> From: Jason Gustafson <jason@confluent.io>
> Sent: Tuesday, December 18, 2018 7:40 AM
> To: dev
> Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> reduce consumer rebalances
>
> Hi Boyang,
>
> Thanks, the KIP looks good. Just one comment.
>
> The new schema for the LeaveGroup request is slightly odd since it is
> handling both the single consumer use case and the administrative use case.
> I wonder we could make it consistent from a batching perspective.
>
> In other words, instead of this:
> LeaveGroupRequest => GroupId MemberId [GroupInstanceId]
>
> Maybe we could do this:
> LeaveGroupRequest => GroupId [GroupInstanceId MemberId]
>
> For dynamic members, GroupInstanceId could be empty, which is consistent
> with JoinGroup. What do you think?
>
> Also, just for clarification, what is the expected behavior if the current
> memberId of a static member is passed to LeaveGroup? Will the static member
> be removed? I know the consumer will not do this, but we'll still have to
> handle the case on the broker.
>
> Best,
> Jason
>
>
> On Mon, Dec 10, 2018 at 11:54 PM Boyang Chen <bchen11@outlook.com> wrote:
>
> > Thanks Stanislav!
> >
> > Get Outlook for iOS<https://aka.ms/o0ukef>
> >
> > ________________________________
> > From: Stanislav Kozlovski <stanislav@confluent.io>
> > Sent: Monday, December 10, 2018 11:28 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> > reduce consumer rebalances
> >
> > This is great work, Boyang. Thank you very much.
> >
> > +1 (non-binding)
> >
> > On Mon, Dec 10, 2018 at 6:09 PM Boyang Chen <bchen11@outlook.com> wrote:
> >
> > > Hey there, could I get more votes on this thread?
> > >
> > > Thanks for the vote from Mayuresh and Mike :)
> > >
> > > Best,
> > > Boyang
> > > ________________________________
> > > From: Mayuresh Gharat <gharatmayuresh15@gmail.com>
> > > Sent: Thursday, December 6, 2018 10:53 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> > > reduce consumer rebalances
> > >
> > > +1 (non-binding)
> > >
> > > Thanks,
> > >
> > > Mayuresh
> > >
> > > On Tue, Dec 4, 2018 at 6:58 AM Mike Freyberger <
> > mike.freyberger@xandr.com>
> > > wrote:
> > >
> > > > +1 (non binding)
> > > >
> > > > On 12/4/18, 9:43 AM, "Patrick Williams" <
> > patrick.williams@storageos.com
> > > >
> > > > wrote:
> > > >
> > > > Pls take me off this VOTE list
> > > >
> > > > Best,
> > > >
> > > > Patrick Williams
> > > >
> > > > Sales Manager, UK & Ireland, Nordics & Israel
> > > > StorageOS
> > > > +44 (0)7549 676279
> > > > patrick.williams@storageos.com
> > > >
> > > > 20 Midtown
> > > > 20 Proctor Street
> > > > Holborn
> > > > London WC1V 6NX
> > > >
> > > > Twitter: @patch37
> > > > LinkedIn: linkedin.com/in/patrickwilliams4 <
> > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flinkedin.com%2Fin%2Fpatrickwilliams4&amp;data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636801101252994092&amp;sdata=ipDTX%2FGARrFkwZfRuOY0M5m3iJ%2Bnkxovv6u9bBDaTyc%3D&amp;reserved=0
> > > >
> > > >
> > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslack.storageos.com%2F&amp;data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636801101252994092&amp;sdata=hxuKU6aZdQU%2FpxpqaaThR6IjpEmwIP5%2F3NhYzMYijkw%3D&amp;reserved=0
> > > >
> > > >
> > > >
> > > > On 03/12/2018, 17:34, "Guozhang Wang" <wangguoz@gmail.com> wrote:
> > > >
> > > > Hello Boyang,
> > > >
> > > > I've browsed through the new wiki and there are still a couple of
> > > > minor
> > > > things to notice:
> > > >
> > > > 1. RemoveMemberFromGroupOptions seems not defined anywhere.
> > > >
> > > > 2. LeaveGroupRequest added a list of group instance id, but still
> > > > keep the
> > > > member id as a singleton; is that intentional? I think to make
> > > the
> > > > protocol
> > > > consistent both member id and instance ids could be plural.
> > > >
> > > > 3. About the *kafka-remove-member-from-group.sh *tool, I'm
> > > > wondering if we
> > > > can defer adding this while just add the corresponding calls of
> > > the
> > > > LeaveGroupRequest inside Streams until we have used it in
> > > > production and
> > > > hence have a better understanding on how flexible or extensible
> > > if
> > > > we want
> > > > to add any cmd tools. The rationale is that if we do not
> > > > necessarily need
> > > > it now, we can always add it later with a more think-through API
> > > > design,
> > > > but if we add the tool in a rush, we may need to extend or modify
> > > > it soon
> > > > after we realize its limits in operations.
> > > >
> > > > Otherwise, I'm +1 on the proposal.
> > > >
> > > > Guozhang
> > > >
> > > >
> > > > On Mon, Dec 3, 2018 at 9:14 AM Boyang Chen <bchen11@outlook.com>
> > > > wrote:
> > > >
> > > > > Hey community friends,
> > > > >
> > > > > after another month of polishing, KIP-345<
> > > > >
> > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&amp;data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636801101252994092&amp;sdata=T4i7L1i0nIeHrrjTeLOOgYKsfzfNEMGDhTazvBEZbXw%3D&amp;reserved=0
> > > > >
> > > > > design is ready for vote. Feel free to add your comment on the
> > > > discussion
> > > > > thread or here.
> > > > >
> > > > > Thanks for your time!
> > > > >
> > > > > Boyang
> > > > > ________________________________
> > > > > From: Boyang Chen <bchen11@outlook.com>
> > > > > Sent: Friday, November 9, 2018 6:35 AM
> > > > > To: dev@kafka.apache.org
> > > > > Subject: [VOTE] KIP-345: Introduce static membership protocol
> > > to
> > > > reduce
> > > > > consumer rebalances
> > > > >
> > > > > Hey all,
> > > > >
> > > > >
> > > > > thanks so much for all the inputs on KIP-345 so far. The
> > > > original proposal
> > > > > has enhanced a lot with your help. To make sure the
> > > > implementation go
> > > > > smoothly without back and forth, I would like to start a vote
> > > on
> > > > the final
> > > > > design agreement now:
> > > > >
> > > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-&amp;data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636801101252994092&amp;sdata=g4%2BMXKpkiQLZXg5HJWfJhw1kc1PbDNwyiX9zkREVqGE%3D&amp;reserved=0
> > > <
> > > > >
> > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&amp;data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636801101252994092&amp;sdata=T4i7L1i0nIeHrrjTeLOOgYKsfzfNEMGDhTazvBEZbXw%3D&amp;reserved=0
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> 345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances<
> > > > >
> > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&amp;data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636801101252994092&amp;sdata=T4i7L1i0nIeHrrjTeLOOgYKsfzfNEMGDhTazvBEZbXw%3D&amp;reserved=0
> > > > > >
> > > > >
> > > > > KIP-345: Introduce static membership protocol to reduce ...<
> > > > >
> > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&amp;data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636801101252994092&amp;sdata=T4i7L1i0nIeHrrjTeLOOgYKsfzfNEMGDhTazvBEZbXw%3D&amp;reserved=0
> > > > > >
> > > > > cwiki.apache.org
> > > > > For stateful applications, one of the biggest performance
> > > > bottleneck is
> > > > > the state shuffling. In Kafka consumer, there is a concept
> > > called
> > > > > "rebalance" which means that for given M partitions and N
> > > > consumers in one
> > > > > consumer group, Kafka will try to balance the load between
> > > > consumers and
> > > > > ideally have ...
> > > > >
> > > > >
> > > > > Let me know if you have any questions.
> > > > >
> > > > >
> > > > > Best,
> > > > >
> > > > > Boyang
> > > > >
> > > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > > --
> > > -Regards,
> > > Mayuresh R. Gharat
> > > (862) 250-7125
> > >
> >
> >
> > --
> > Best,
> > Stanislav
> >
>


-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message