kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id
Date Fri, 27 Jul 2018 06:09:42 GMT
Hi Boyang,

Thanks for the proposed KIP. I made a pass over the wiki and here are some
comments / questions:

1. In order to preserve broker compatibility, we need to make sure the
broker version discovery logic can be integrated with this new logic. I.e.
if a newer versioned consumer is talking to an older versioned broker who
does not recognize V4, the client needs to downgrade its JoinGroupRequest
version to V3 and not setting the member-id specifically. You can take a
look at the ApiVersionsRequest and see how to work with it.

2. There may exist some manners to validate that two different clients do
not send with the same member id, for example if we pass along the
host:port information from KafkaApis to the GroupCoordinator interface. But
I think this is overly complicated the logic and may not worthwhile than
relying on users to specify unique member ids.

3. Minor: you would need to bumping up the version of JoinGroupResponse to
V4 as well.

4. Minor: in the wiki page, you need to specify the actual string value for
`MEMBER_ID`, for example "member.id".

5. When this additional config it specified by users, we should consider
setting the default of internal `LEAVE_GROUP_ON_CLOSE_CONFIG` to false,
since otherwise its effectiveness would be less.


Guozhang



On Thu, Jul 26, 2018 at 9:20 PM, Boyang Chen <bchen11@outlook.com> wrote:

> Hey friends,
>
>
> I would like to open a discussion thread on KIP-345:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A
> +Reduce+multiple+consumer+rebalances+by+specifying+member+id
>
>
> This KIP is trying to resolve multiple rebalances by maintaining the
> consumer member id across rebalance generations. I have verified the theory
> on our internal Stream application, and it could reduce rebalance time to a
> few seconds when service is rolling restart.
>
>
> Let me know your thoughts, thank you!
>
>
> Best,
>
> Boyang
>



-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message