kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Damian Guy <damian....@gmail.com>
Subject Re: [VOTE] KIP-134: Delay initial consumer group rebalance
Date Wed, 07 Jun 2017 09:26:08 GMT
Hi Jun/Ismael,

Sounds good to me.

Thanks,
Damian

On Tue, 6 Jun 2017 at 23:08 Ismael Juma <ismael@juma.me.uk> wrote:

> Hi Jun,
>
> The console consumer issue also came up in a conversation I was having
> recently. Seems like the config/server.properties change is a reasonable
> compromise given that we have other defaults that are for development.
>
> Ismael
>
> On Tue, Jun 6, 2017 at 10:59 PM, Jun Rao <jun@confluent.io> wrote:
>
> > Hi, Everyone,
> >
> > Sorry for being late on this thread. I just came across this thread. I
> have
> > a couple of concerns on this. (1) It seems the amount of delay will be
> > application specific. So, it seems that it's better for the delay to be a
> > client side config instead of a server side one? (2) When running console
> > consumer in quickstart, a minimum of 3 sec delay seems to be a bad
> > experience for our users.
> >
> > Since we are getting late into the release cycle, it may be a bit too
> late
> > to make big changes in the 0.11 release. Perhaps we should at least
> > consider overriding the delay in config/server.properties to 0 to improve
> > the quickstart experience?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Tue, Apr 11, 2017 at 12:19 AM, Damian Guy <damian.guy@gmail.com>
> wrote:
> >
> > > Hi Onur,
> > >
> > > It was in my previous email. But here it is again.
> > >
> > > ============================================================
> > >
> > > 1. Better rebalance timing. We will try to rebalance only when all the
> > > consumers in a group have joined. The challenge would be someone has to
> > > define what does ALL consumers mean, it could either be a time or
> number
> > of
> > > consumers, etc.
> > >
> > > 2. Avoid frequent rebalance. For example, if there are 100 consumers
> in a
> > > group, today, in the worst case, we may end up with 100 rebalances even
> > if
> > > all the consumers joined the group in a reasonably small amount of
> time.
> > > Frequent rebalance is also a bad thing for brokers.
> > >
> > > Having a client side configuration may solve problem 1 better because
> > each
> > > consumer group can potentially configure their own timing. However, it
> > does
> > > not really prevent frequent rebalance in general because some of the
> > > consumers can be misconfigured. (This may have something to do with
> > KIP-124
> > > as well. But if quota is applied on the JoinGroup/SyncGroup request it
> > may
> > > cause some unwanted cascading effects.)
> > >
> > > Having a broker side configuration may result in less flexibility for
> > each
> > > consumer group, but it can prevent frequent rebalance better. I think
> > with
> > > some reasonable design, the rebalance timing issue can be resolved on
> the
> > > broker side as well. Matthias had a good point on extending the delay
> > when
> > > a new consumer joins a group (we actually did something similar to
> batch
> > > ISR change propagation). For example, let's say on the broker side, we
> > will
> > > always delay 2 seconds each time we see a new consumer joining a
> consumer
> > > group. This would probably work for most of the consumer groups and
> will
> > > also limit the rebalance frequency to protect the brokers.
> > >
> > > I am not sure about the streams use case here, but if something like 2
> > > seconds of delay is acceptable for streams, I would prefer adding the
> > > configuration to the broker so that we can address both problems.
> > >
> > > On Thu, 6 Apr 2017 at 17:11 Onur Karaman <onurkaraman.apache@gmail.com
> >
> > > wrote:
> > >
> > > > Hi Damian.
> > > >
> > > > Can you copy the point Becket made earlier that you say isn't
> > addressed?
> > > >
> > > > On Thu, Apr 6, 2017 at 2:51 AM, Damian Guy <damian.guy@gmail.com>
> > wrote:
> > > >
> > > > > Thanks all, the Vote is now closed and the KIP has been accepted
> > with 9
> > > > +1s
> > > > >
> > > > > 3 binding::
> > > > > Guozhang,
> > > > > Jason,
> > > > > Ismael
> > > > >
> > > > > 6 non-binding:
> > > > > Bill,
> > > > > Eno,
> > > > > Mathieu,
> > > > > Matthias,
> > > > > Dong,
> > > > > Mickael
> > > > >
> > > > > Thanks,
> > > > > Damian
> > > > >
> > > > > On Thu, 6 Apr 2017 at 09:26 Ismael Juma <ismael@juma.me.uk>
wrote:
> > > > >
> > > > > > Thanks for the KIP, +1 (binding).
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Thu, Mar 30, 2017 at 8:55 PM, Jason Gustafson <
> > jason@confluent.io
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > +1 Thanks for the KIP!
> > > > > > >
> > > > > > > On Thu, Mar 30, 2017 at 12:51 PM, Guozhang Wang <
> > > wangguoz@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > +1
> > > > > > > >
> > > > > > > > Sorry about the previous email, Gmail seems be collapsing
> them
> > > > into a
> > > > > > > > single thread on my inbox.
> > > > > > > >
> > > > > > > > Guozhang
> > > > > > > >
> > > > > > > > On Thu, Mar 30, 2017 at 11:34 AM, Guozhang Wang <
> > > > wangguoz@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Damian, could you create a new thread for the
voting
> process?
> > > > > > > > >
> > > > > > > > > Thanks!
> > > > > > > > >
> > > > > > > > > Guozhang
> > > > > > > > >
> > > > > > > > > On Thu, Mar 30, 2017 at 10:33 AM, Bill Bejeck
<
> > > bbejeck@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> +1(non-binding)
> > > > > > > > >>
> > > > > > > > >> On Thu, Mar 30, 2017 at 1:30 PM, Eno Thereska
<
> > > > > > eno.thereska@gmail.com
> > > > > > > >
> > > > > > > > >> wrote:
> > > > > > > > >>
> > > > > > > > >> > +1 (non binding)
> > > > > > > > >> >
> > > > > > > > >> > Thanks
> > > > > > > > >> > Eno
> > > > > > > > >> > > On 30 Mar 2017, at 18:01, Matthias
J. Sax <
> > > > > > matthias@confluent.io>
> > > > > > > > >> wrote:
> > > > > > > > >> > >
> > > > > > > > >> > > +1
> > > > > > > > >> > >
> > > > > > > > >> > > On 3/30/17 3:46 AM, Damian Guy
wrote:
> > > > > > > > >> > >> Hi All,
> > > > > > > > >> > >>
> > > > > > > > >> > >> I'd like to start the voting
thread on KIP-134:
> > > > > > > > >> > >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > > > >> > 134%3A+Delay+initial+consumer+group+rebalance
> > > > > > > > >> > >>
> > > > > > > > >> > >> Thanks,
> > > > > > > > >> > >> Damian
> > > > > > > > >> > >>
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > -- Guozhang
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > -- Guozhang
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message