kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Grant Henke <ghe...@cloudera.com>
Subject Re: Consider increasing the default reserved.broker.max.id
Date Fri, 08 Jan 2016 22:18:07 GMT
I agree that many people id their brokers differently and increasing the
default will only handle a subset of those schemes. Though I think
increasing it to some reasonable value may help decrease issues drastically

I also think some longer term fix that avoids collisions all together would
be nice. Though I am not sure what that long term solution is. We would
need to introduce something that a configured broker id is not allowed to
set. Any ideas?

I also wanted to note here that while investigating this I found some
interesting special cases/rules for the reserved.broker.max.id config.

1. Because a zookeeper sequence value is added to that value to generate
the unique id's, the value once configured and used *cannot be decreased*
and still guaranteeing no collisions.

2. Because the id was generated it can never be manually set in the config.
Therefore if you need to stand up a new machine with the same broker ids
(perhaps for recovery) you can't set this value manually. The workaround
would be to set the value in the meta.properties file of all the log
directories. (note: I haven't fully vetted this yet)

On Wed, Dec 23, 2015 at 5:25 PM, Ewen Cheslack-Postava <ewen@confluent.io>

> Which other numbering schemes do we want to be able to un-break by
> increasing this default? For example, I know some people use the IP address
> with dots removed -- we'd have to use a very large # to make sure that
> worked. Before making another change, it'd be good to know what other
> schemes people are using and that we'd really be fixing the issue for
> someone.
> -Ewen
> On Fri, Dec 18, 2015 at 9:37 AM, Ismael Juma <ismael@juma.me.uk> wrote:
> > On Fri, Dec 18, 2015 at 4:44 PM, Grant Henke <ghenke@cloudera.com>
> wrote:
> >
> > > There is some discussion on KAFKA-1070
> > > <https://issues.apache.org/jira/browse/KAFKA-1070> around the design
> > > choice
> > > and compatibility. The value 1000 was thrown out as a quick example but
> > it
> > > was never discussed beyond that. The discussion also sites a few cases
> > > where a value of 1000 would cause issue.
> > >
> >
> > Thanks for digging that up. Also worth noting that Jay said:
> >
> > "I think we can get around the problem you point out by just defaulting
> the
> > node id sequence to 1000. This could theoretically conflict but most
> people
> > number from 0 or 1 and we can discuss this in the release notes. Our plan
> > will be to release with support for both configured node ids and assigned
> > node ids for compatibility. After a couple of releases we will remove the
> > config."
> >
> > Ismael
> >
> --
> Thanks,
> Ewen

Grant Henke
Software Engineer | Cloudera
grant@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message