kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Otto <o...@wikimedia.org>
Subject Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation
Date Thu, 23 Aug 2018 15:08:59 GMT
Yup :)

On Thu, Aug 23, 2018 at 11:04 AM Ismael Juma <ismael@juma.me.uk> wrote:

> Andrew, one question: you are relying on auto topic creation for the
> producer and that's why you can't just disable it?
>
> On Thu, Aug 23, 2018 at 8:01 AM Ismael Juma <ismael@juma.me.uk> wrote:
>
> > Thanks for sharing Andrew!
> >
> > Ismael
> >
> > On Thu, Aug 23, 2018 at 7:57 AM Andrew Otto <otto@wikimedia.org> wrote:
> >
> >> We recently had a pretty serious Kafka outage
> >> <
> >>
> https://wikitech.wikimedia.org/wiki/Incident_documentation/20180711-kafka-eqiad#Summary
> >> >
> >> caused by a bug in one of our consumers that caused it to create new
> >> topics
> >> in an infinite loop AKA a topic bomb!  Having consumers restricted from
> >> creating topics would have prevented this for us.
> >>
> >> On Thu, Aug 23, 2018 at 4:27 AM Ismael Juma <ismael@juma.me.uk> wrote:
> >>
> >> > Generally, I think positive configs (`allow` instead of `suppress`)
> are
> >> > easier to understand.
> >> >
> >> > Ismael
> >> >
> >> > On Wed, Aug 22, 2018 at 11:05 PM Matthias J. Sax <
> matthias@confluent.io
> >> >
> >> > wrote:
> >> >
> >> > > Thanks for the summary!
> >> > >
> >> > > We might want to add a diagram/table to the docs when we add this
> >> > > feature (with whatever config name we choose) to explain how broker
> >> > > config `auto.create.topics.enable` and the consumer config work
> >> together.
> >> > >
> >> > > I think both options are equally easy to understand. "allow" means
> >> > > follow the broker config, while "suppress" implies ignore the broker
> >> > > config and don't auto-create.
> >> > >
> >> > >
> >> > > -Matthias
> >> > >
> >> > >
> >> > > On 8/22/18 10:36 PM, Dhruvil Shah wrote:
> >> > > > *"suppress" is the opposite of "allow", so
> >> > > > setting suppress.auto.create.topics=false would mean that we
do
> >> _not_
> >> > > allow
> >> > > > auto topic creation; when set to true, the server configuration
> will
> >> > > > determine whether we allow automatic creation or not.*
> >> > > >
> >> > > > Sorry, I meant suppress.auto.create.topics=true above to disallow
> >> auto
> >> > > > topic creation.
> >> > > >
> >> > > >
> >> > > > On Wed, Aug 22, 2018 at 10:34 PM Dhruvil Shah <
> dhruvil@confluent.io
> >> >
> >> > > wrote:
> >> > > >
> >> > > >> To be clear, we will allow auto topic creation only when
server
> >> config
> >> > > >> auto.create.topics.enable=true and consumer config
> >> > > >> allow.auto.create.topics=true; when either is false, we would
not
> >> > create
> >> > > >> the topic if it does not exist.
> >> > > >>
> >> > > >> "suppress" is the opposite of "allow", so
> >> > > >> setting suppress.auto.create.topics=false would mean that
we do
> >> _not_
> >> > > allow
> >> > > >> auto topic creation; when set to true, the server configuration
> >> will
> >> > > >> determine whether we allow automatic creation or not.
> >> > > >>
> >> > > >> I think "allow" is easier to understand but I am open to
> >> suggestions.
> >> > > >>
> >> > > >> - Dhruvil
> >> > > >>
> >> > > >> On Wed, Aug 22, 2018 at 6:53 PM Brandon Kirchner <
> >> > > >> brandon.kirchner@gmail.com> wrote:
> >> > > >>
> >> > > >>> “allow=false” seems a bit more intuitive to me than
> >> “suppress=false”
> >> > > >>>
> >> > > >>> Brandon
> >> > > >>>
> >> > > >>>> On Aug 22, 2018, at 8:48 PM, Ted Yu <yuzhihong@gmail.com>
> wrote:
> >> > > >>>>
> >> > > >>>> We may also consider :
> >> > > >>>>
> >> > > >>>> "suppress.auto.topic.creation"
> >> > > >>>>
> >> > > >>>> or
> >> > > >>>>
> >> > > >>>> "allow.auto.topic.creation"
> >> > > >>>>
> >> > > >>>> w.r.t. suppress or allow, I don't have strong opinion
either.
> >> It's
> >> > > just
> >> > > >>> a
> >> > > >>>> matter of choosing the proper default value.
> >> > > >>>>
> >> > > >>>> Cheers
> >> > > >>>>
> >> > > >>>>> On Wed, Aug 22, 2018 at 6:00 PM Dhruvil Shah
<
> >> dhruvil@confluent.io
> >> > >
> >> > > >>> wrote:
> >> > > >>>>>
> >> > > >>>>> Hi Matthias,
> >> > > >>>>>
> >> > > >>>>> Do you mean something like "suppress.auto.create.topic"?
I am
> >> > leaning
> >> > > >>> a bit
> >> > > >>>>> towards "allow.auto.create.topics" but I don't
have a strong
> >> > > preference
> >> > > >>>>> either. Let's wait to hear if anyone else has
an opinion on
> >> this.
> >> > > >>>>>
> >> > > >>>>> Thanks,
> >> > > >>>>> Dhruvil
> >> > > >>>>>
> >> > > >>>>> On Tue, Aug 21, 2018 at 5:28 PM Matthias J. Sax
<
> >> > > matthias@confluent.io
> >> > > >>>>
> >> > > >>>>> wrote:
> >> > > >>>>>
> >> > > >>>>>> Thanks for the KIP Dhruvil!
> >> > > >>>>>>
> >> > > >>>>>> I agree with Jason's comment. An alternative
might be to use
> >> > > >>> "suppress"
> >> > > >>>>>> what would revert the logic of "allow". Not
sure which one is
> >> more
> >> > > >>>>>> intuitive and I am fine with both (no personal
preference).
> >> Just
> >> > > >>> wanted
> >> > > >>>>>> to mention it as an alternative.
> >> > > >>>>>>
> >> > > >>>>>> Don't have any further comments/question
so far.
> >> > > >>>>>>
> >> > > >>>>>>
> >> > > >>>>>> -Matthias
> >> > > >>>>>>
> >> > > >>>>>>
> >> > > >>>>>>
> >> > > >>>>>>> On 8/21/18 4:42 PM, Jason Gustafson wrote:
> >> > > >>>>>>> Hey Dhruvil,
> >> > > >>>>>>>
> >> > > >>>>>>> I would suggest using the verb "allow"
rather than "enable.
> >> The
> >> > > >>>>> consumer
> >> > > >>>>>>> cannot enable auto topic creation because
it is configured
> on
> >> the
> >> > > >>>>> broker.
> >> > > >>>>>>> All it can do is prevent it from happening
if it is enabled.
> >> > > >>>>>>>
> >> > > >>>>>>> -Jason
> >> > > >>>>>>>
> >> > > >>>>>>> On Tue, Aug 21, 2018 at 3:56 PM, Dhruvil
Shah <
> >> > > dhruvil@confluent.io>
> >> > > >>>>>> wrote:
> >> > > >>>>>>>
> >> > > >>>>>>>> Hi,
> >> > > >>>>>>>>
> >> > > >>>>>>>> I would like to start discussion
on KIP-361 that proposes
> we
> >> > add a
> >> > > >>>>>> consumer
> >> > > >>>>>>>> configuration to disable auto topic
creation.
> >> > > >>>>>>>>
> >> > > >>>>>>>> Link to the KIP:
> >> > > >>>>>>>>
> >> > > >>>>>>
> >> > > >>>>>
> >> > > >>>
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+
> >> > > >>>>>>>> Configuration+to+Disable+Auto+Topic+Creation
> >> > > >>>>>>>>
> >> > > >>>>>>>> Suggestions and feedback are welcome!
> >> > > >>>>>>>>
> >> > > >>>>>>>> Thanks,
> >> > > >>>>>>>> Dhruvil
> >> > > >>>>>>>>
> >> > > >>>>>>>
> >> > > >>>>>>
> >> > > >>>>>>
> >> > > >>>>>
> >> > > >>>
> >> > > >>
> >> > > >
> >> > >
> >> > >
> >> >
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message