kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Otto <o...@wikimedia.org>
Subject Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation
Date Thu, 23 Aug 2018 14:57:32 GMT
We recently had a pretty serious Kafka outage
<https://wikitech.wikimedia.org/wiki/Incident_documentation/20180711-kafka-eqiad#Summary>
caused by a bug in one of our consumers that caused it to create new topics
in an infinite loop AKA a topic bomb!  Having consumers restricted from
creating topics would have prevented this for us.

On Thu, Aug 23, 2018 at 4:27 AM Ismael Juma <ismael@juma.me.uk> wrote:

> Generally, I think positive configs (`allow` instead of `suppress`) are
> easier to understand.
>
> Ismael
>
> On Wed, Aug 22, 2018 at 11:05 PM Matthias J. Sax <matthias@confluent.io>
> wrote:
>
> > Thanks for the summary!
> >
> > We might want to add a diagram/table to the docs when we add this
> > feature (with whatever config name we choose) to explain how broker
> > config `auto.create.topics.enable` and the consumer config work together.
> >
> > I think both options are equally easy to understand. "allow" means
> > follow the broker config, while "suppress" implies ignore the broker
> > config and don't auto-create.
> >
> >
> > -Matthias
> >
> >
> > On 8/22/18 10:36 PM, Dhruvil Shah wrote:
> > > *"suppress" is the opposite of "allow", so
> > > setting suppress.auto.create.topics=false would mean that we do _not_
> > allow
> > > auto topic creation; when set to true, the server configuration will
> > > determine whether we allow automatic creation or not.*
> > >
> > > Sorry, I meant suppress.auto.create.topics=true above to disallow auto
> > > topic creation.
> > >
> > >
> > > On Wed, Aug 22, 2018 at 10:34 PM Dhruvil Shah <dhruvil@confluent.io>
> > wrote:
> > >
> > >> To be clear, we will allow auto topic creation only when server config
> > >> auto.create.topics.enable=true and consumer config
> > >> allow.auto.create.topics=true; when either is false, we would not
> create
> > >> the topic if it does not exist.
> > >>
> > >> "suppress" is the opposite of "allow", so
> > >> setting suppress.auto.create.topics=false would mean that we do _not_
> > allow
> > >> auto topic creation; when set to true, the server configuration will
> > >> determine whether we allow automatic creation or not.
> > >>
> > >> I think "allow" is easier to understand but I am open to suggestions.
> > >>
> > >> - Dhruvil
> > >>
> > >> On Wed, Aug 22, 2018 at 6:53 PM Brandon Kirchner <
> > >> brandon.kirchner@gmail.com> wrote:
> > >>
> > >>> “allow=false” seems a bit more intuitive to me than “suppress=false”
> > >>>
> > >>> Brandon
> > >>>
> > >>>> On Aug 22, 2018, at 8:48 PM, Ted Yu <yuzhihong@gmail.com>
wrote:
> > >>>>
> > >>>> We may also consider :
> > >>>>
> > >>>> "suppress.auto.topic.creation"
> > >>>>
> > >>>> or
> > >>>>
> > >>>> "allow.auto.topic.creation"
> > >>>>
> > >>>> w.r.t. suppress or allow, I don't have strong opinion either. It's
> > just
> > >>> a
> > >>>> matter of choosing the proper default value.
> > >>>>
> > >>>> Cheers
> > >>>>
> > >>>>> On Wed, Aug 22, 2018 at 6:00 PM Dhruvil Shah <dhruvil@confluent.io
> >
> > >>> wrote:
> > >>>>>
> > >>>>> Hi Matthias,
> > >>>>>
> > >>>>> Do you mean something like "suppress.auto.create.topic"? I
am
> leaning
> > >>> a bit
> > >>>>> towards "allow.auto.create.topics" but I don't have a strong
> > preference
> > >>>>> either. Let's wait to hear if anyone else has an opinion on
this.
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Dhruvil
> > >>>>>
> > >>>>> On Tue, Aug 21, 2018 at 5:28 PM Matthias J. Sax <
> > matthias@confluent.io
> > >>>>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Thanks for the KIP Dhruvil!
> > >>>>>>
> > >>>>>> I agree with Jason's comment. An alternative might be to
use
> > >>> "suppress"
> > >>>>>> what would revert the logic of "allow". Not sure which
one is more
> > >>>>>> intuitive and I am fine with both (no personal preference).
Just
> > >>> wanted
> > >>>>>> to mention it as an alternative.
> > >>>>>>
> > >>>>>> Don't have any further comments/question so far.
> > >>>>>>
> > >>>>>>
> > >>>>>> -Matthias
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>> On 8/21/18 4:42 PM, Jason Gustafson wrote:
> > >>>>>>> Hey Dhruvil,
> > >>>>>>>
> > >>>>>>> I would suggest using the verb "allow" rather than
"enable. The
> > >>>>> consumer
> > >>>>>>> cannot enable auto topic creation because it is configured
on the
> > >>>>> broker.
> > >>>>>>> All it can do is prevent it from happening if it is
enabled.
> > >>>>>>>
> > >>>>>>> -Jason
> > >>>>>>>
> > >>>>>>> On Tue, Aug 21, 2018 at 3:56 PM, Dhruvil Shah <
> > dhruvil@confluent.io>
> > >>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> Hi,
> > >>>>>>>>
> > >>>>>>>> I would like to start discussion on KIP-361 that
proposes we
> add a
> > >>>>>> consumer
> > >>>>>>>> configuration to disable auto topic creation.
> > >>>>>>>>
> > >>>>>>>> Link to the KIP:
> > >>>>>>>>
> > >>>>>>
> > >>>>>
> > >>>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+
> > >>>>>>>> Configuration+to+Disable+Auto+Topic+Creation
> > >>>>>>>>
> > >>>>>>>> Suggestions and feedback are welcome!
> > >>>>>>>>
> > >>>>>>>> Thanks,
> > >>>>>>>> Dhruvil
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>
> > >>
> > >
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message