kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stanislav Kozlovski <stanis...@confluent.io>
Subject Re: [Discuss] KIP-389: Enforce group.max.size to cap member metadata growth
Date Mon, 26 Nov 2018 10:14:04 GMT
Hey everybody,

It's been a week since this KIP and not much discussion has been made.
I assume that this is a straight forward change and I will open a voting
thread in the next couple of days if nobody has anything to suggest.

Best,
Stanislav

On Thu, Nov 22, 2018 at 12:56 PM Stanislav Kozlovski <stanislav@confluent.io>
wrote:

> Greetings everybody,
>
> I have enriched the KIP a bit with a bigger Motivation section and also
> renamed it.
> KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-389%3A+Introduce+a+configurable+consumer+group+size+limit
>
> I'm looking forward to discussions around it.
>
> Best,
> Stanislav
>
> On Tue, Nov 20, 2018 at 1:47 PM Stanislav Kozlovski <
> stanislav@confluent.io> wrote:
>
>> Hey there everybody,
>>
>> Thanks for the introduction Boyang. I appreciate the effort you are
>> putting into improving consumer behavior in Kafka.
>>
>> @Matt
>> I also believe the default value is high. In my opinion, we should aim to
>> a default cap around 250. This is because in the current model any consumer
>> rebalance is disrupting to every consumer. The bigger the group, the longer
>> this period of disruption.
>>
>> If you have such a large consumer group, chances are that your
>> client-side logic could be structured better and that you are not using the
>> high number of consumers to achieve high throughput.
>> 250 can still be considered of a high upper bound, I believe in practice
>> users should aim to not go over 100 consumers per consumer group.
>>
>> In regards to the cap being global/per-broker, I think that we should
>> consider whether we want it to be global or *per-topic*. For the time
>> being, I believe that having it per-topic with a global default might be
>> the best situation. Having it global only seems a bit restricting to me and
>> it never hurts to support more fine-grained configurability (given it's the
>> same config, not a new one being introduced).
>>
>> On Tue, Nov 20, 2018 at 11:32 AM Boyang Chen <bchen11@outlook.com> wrote:
>>
>>> Thanks Matt for the suggestion! I'm still open to any suggestion to
>>> change the default value. Meanwhile I just want to point out that this
>>> value is a just last line of defense, not a real scenario we would expect.
>>>
>>>
>>> In the meanwhile, I discussed with Stanislav and he would be driving the
>>> 389 effort from now on. Stanislav proposed the idea in the first place and
>>> had already come up a draft design, while I will keep focusing on KIP-345
>>> effort to ensure solving the edge case described in the JIRA<
>>> https://issues.apache.org/jira/browse/KAFKA-7610>.
>>>
>>>
>>> Thank you Stanislav for making this happen!
>>>
>>>
>>> Boyang
>>>
>>> ________________________________
>>> From: Matt Farmer <matt@frmr.me>
>>> Sent: Tuesday, November 20, 2018 10:24 AM
>>> To: dev@kafka.apache.org
>>> Subject: Re: [Discuss] KIP-389: Enforce group.max.size to cap member
>>> metadata growth
>>>
>>> Thanks for the KIP.
>>>
>>> Will this cap be a global cap across the entire cluster or per broker?
>>>
>>> Either way the default value seems a bit high to me, but that could just
>>> be
>>> from my own usage patterns. I’d have probably started with 500 or 1k but
>>> could be easily convinced that’s wrong.
>>>
>>> Thanks,
>>> Matt
>>>
>>> On Mon, Nov 19, 2018 at 8:51 PM Boyang Chen <bchen11@outlook.com> wrote:
>>>
>>> > Hey folks,
>>> >
>>> >
>>> > I would like to start a discussion on KIP-389:
>>> >
>>> >
>>> >
>>> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-389%253A%2BEnforce%2Bgroup.max.size%2Bto%2Bcap%2Bmember%2Bmetadata%2Bgrowth&amp;data=02%7C01%7C%7Cb0ee4fe97ad44cc046eb08d64e8f5d90%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636782774981237462&amp;sdata=Q2T7hIoVq8GiPVhr0HIxVkGNChkiz1Pvk2zyLD5gCu8%3D&amp;reserved=0
>>> >
>>> >
>>> > This is a pretty simple change to cap the consumer group size for
>>> broker
>>> > stability. Give me your valuable feedback when you got time.
>>> >
>>> >
>>> > Thank you!
>>> >
>>>
>>
>>
>> --
>> Best,
>> Stanislav
>>
>
>
> --
> Best,
> Stanislav
>


-- 
Best,
Stanislav

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message