kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances
Date Tue, 04 Dec 2018 07:21:41 GMT
2. That's a good point. I think I'm convinced.


Guozhang

On Mon, Dec 3, 2018 at 5:43 PM Boyang Chen <bchen11@outlook.com> wrote:

> Thanks Guozhang for the reply!
>
> 1. RemoveMemberFromGroupOptions seems not defined anywhere.
> Added the definition.
> 2. LeaveGroupRequest added a list of group instance id, but still keep the
> member id as a singleton; is that intentional? I think to make the protocol
> consistent both member id and instance ids could be plural.
> Since a dynamic member would send LeaveGroupRequest with its member.id,
> I feel it's ok to keep the existing API instead of expanding singleton to
> a list. Haven't
> been able to define a scenario where we need to pass a list of `member.id
> `.
> What do you think?
>
> 3. About the *kafka-remove-member-from-group.sh *tool, I'm wondering if we
> can defer adding this while just add the corresponding calls of the
> LeaveGroupRequest inside Streams until we have used it in production and
> hence have a better understanding on how flexible or extensible if we want
> to add any cmd tools. The rationale is that if we do not necessarily need
> it now, we can always add it later with a more think-through API design,
> but if we add the tool in a rush, we may need to extend or modify it soon
> after we realize its limits in operations.
> Totally agree. I moved this part to the future work, because tooling
> options could be addressed
> in a separate KIP and a universally favorable solution could be discussed
> independently (for different
> company setup)
>
> Best,
> Boyang
>
> ________________________________
> From: Guozhang Wang <wangguoz@gmail.com>
> Sent: Tuesday, December 4, 2018 1:27 AM
> To: dev
> Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> reduce consumer rebalances
>
> Hello Boyang,
>
> I've browsed through the new wiki and there are still a couple of minor
> things to notice:
>
> 1. RemoveMemberFromGroupOptions seems not defined anywhere.
>
> 2. LeaveGroupRequest added a list of group instance id, but still keep the
> member id as a singleton; is that intentional? I think to make the protocol
> consistent both member id and instance ids could be plural.
>
> 3. About the *kafka-remove-member-from-group.sh *tool, I'm wondering if we
> can defer adding this while just add the corresponding calls of the
> LeaveGroupRequest inside Streams until we have used it in production and
> hence have a better understanding on how flexible or extensible if we want
> to add any cmd tools. The rationale is that if we do not necessarily need
> it now, we can always add it later with a more think-through API design,
> but if we add the tool in a rush, we may need to extend or modify it soon
> after we realize its limits in operations.
>
> Otherwise, I'm +1 on the proposal.
>
> Guozhang
>
>
> On Mon, Dec 3, 2018 at 9:14 AM Boyang Chen <bchen11@outlook.com> wrote:
>
> > Hey community friends,
> >
> > after another month of polishing, KIP-345<
> >
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&amp;data=02%7C01%7C%7C94b56c977f3647e1141908d659458c8c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636794552568572008&amp;sdata=LiNnhFJm8Avri26aEBa3q4%2Fr4aRKVIrZzKHzn71U3Xk%3D&amp;reserved=0
> >
> > design is ready for vote. Feel free to add your comment on the discussion
> > thread or here.
> >
> > Thanks for your time!
> >
> > Boyang
> > ________________________________
> > From: Boyang Chen <bchen11@outlook.com>
> > Sent: Friday, November 9, 2018 6:35 AM
> > To: dev@kafka.apache.org
> > Subject: [VOTE] KIP-345: Introduce static membership protocol to reduce
> > consumer rebalances
> >
> > Hey all,
> >
> >
> > thanks so much for all the inputs on KIP-345 so far. The original
> proposal
> > has enhanced a lot with your help. To make sure the implementation go
> > smoothly without back and forth, I would like to start a vote on the
> final
> > design agreement now:
> >
> >
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-&amp;data=02%7C01%7C%7C94b56c977f3647e1141908d659458c8c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636794552568572008&amp;sdata=GwbfkDFkY2m38V2e%2B6bEWU7PKWPoia5Hw6KmdOXrdcs%3D&amp;reserved=0
> <
> >
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&amp;data=02%7C01%7C%7C94b56c977f3647e1141908d659458c8c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636794552568572008&amp;sdata=LiNnhFJm8Avri26aEBa3q4%2Fr4aRKVIrZzKHzn71U3Xk%3D&amp;reserved=0
> > >
> >
> >
> 345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances<
> >
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&amp;data=02%7C01%7C%7C94b56c977f3647e1141908d659458c8c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636794552568572008&amp;sdata=LiNnhFJm8Avri26aEBa3q4%2Fr4aRKVIrZzKHzn71U3Xk%3D&amp;reserved=0
> > >
> >
> > KIP-345: Introduce static membership protocol to reduce ...<
> >
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&amp;data=02%7C01%7C%7C94b56c977f3647e1141908d659458c8c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636794552568572008&amp;sdata=LiNnhFJm8Avri26aEBa3q4%2Fr4aRKVIrZzKHzn71U3Xk%3D&amp;reserved=0
> > >
> > cwiki.apache.org
> > For stateful applications, one of the biggest performance bottleneck is
> > the state shuffling. In Kafka consumer, there is a concept called
> > "rebalance" which means that for given M partitions and N consumers in
> one
> > consumer group, Kafka will try to balance the load between consumers and
> > ideally have ...
> >
> >
> > Let me know if you have any questions.
> >
> >
> > Best,
> >
> > Boyang
> >
> >
>
> --
> -- Guozhang
>


-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message