kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <matth...@confluent.io>
Subject Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation
Date Fri, 07 Sep 2018 04:52:49 GMT
What is the status of this KIP?

I think you can start a VOTE Dhruvil.


-Matthias

On 8/23/18 9:52 AM, Ismael Juma wrote:
> Yeah, the reason why we want to deprecate the auto create functionality is
> that it happens when a metadata request is done instead of when a write
> operation happens. So, there's no reason to differentiate between the two.
> 
> Ismael
> 
> On Thu, Aug 23, 2018 at 8:16 AM Andrew Otto <otto@wikimedia.org> wrote:
> 
>> Ah, I just realized that as proposed this is only for the Java consumer
>> client, correct?  Would it be possible to make this a broker config, like
>> the current one?  Something like:
>>
>> auto.create.topics.enable=true # allow both producer and consumer to create
>> auto.create.topics.enable=consumer # allow only consumer to create
>> auto.create.topics.enable=producer # allow only producer to create
>> auto.create.topics.enable=false # deny any auto topic creation
>>
>> Perhaps the broker doesn’t differentiate between the type of client
>> connection. If not, I guess this wouldn’t be possible.
>>
>>
>>
>> On Thu, Aug 23, 2018 at 11:08 AM Andrew Otto <otto@wikimedia.org> wrote:
>>
>>> Yup :)
>>>
>>> On Thu, Aug 23, 2018 at 11:04 AM Ismael Juma <ismael@juma.me.uk> wrote:
>>>
>>>> Andrew, one question: you are relying on auto topic creation for the
>>>> producer and that's why you can't just disable it?
>>>>
>>>> On Thu, Aug 23, 2018 at 8:01 AM Ismael Juma <ismael@juma.me.uk> wrote:
>>>>
>>>>> Thanks for sharing Andrew!
>>>>>
>>>>> Ismael
>>>>>
>>>>> On Thu, Aug 23, 2018 at 7:57 AM Andrew Otto <otto@wikimedia.org>
>> wrote:
>>>>>
>>>>>> We recently had a pretty serious Kafka outage
>>>>>> <
>>>>>>
>>>>
>> https://wikitech.wikimedia.org/wiki/Incident_documentation/20180711-kafka-eqiad#Summary
>>>>>>>
>>>>>> caused by a bug in one of our consumers that caused it to create
new
>>>>>> topics
>>>>>> in an infinite loop AKA a topic bomb!  Having consumers restricted
>> from
>>>>>> creating topics would have prevented this for us.
>>>>>>
>>>>>> On Thu, Aug 23, 2018 at 4:27 AM Ismael Juma <ismael@juma.me.uk>
>> wrote:
>>>>>>
>>>>>>> Generally, I think positive configs (`allow` instead of `suppress`)
>>>> are
>>>>>>> easier to understand.
>>>>>>>
>>>>>>> Ismael
>>>>>>>
>>>>>>> On Wed, Aug 22, 2018 at 11:05 PM Matthias J. Sax <
>>>> matthias@confluent.io
>>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks for the summary!
>>>>>>>>
>>>>>>>> We might want to add a diagram/table to the docs when we
add this
>>>>>>>> feature (with whatever config name we choose) to explain
how
>> broker
>>>>>>>> config `auto.create.topics.enable` and the consumer config
work
>>>>>> together.
>>>>>>>>
>>>>>>>> I think both options are equally easy to understand. "allow"
>> means
>>>>>>>> follow the broker config, while "suppress" implies ignore
the
>>>> broker
>>>>>>>> config and don't auto-create.
>>>>>>>>
>>>>>>>>
>>>>>>>> -Matthias
>>>>>>>>
>>>>>>>>
>>>>>>>> On 8/22/18 10:36 PM, Dhruvil Shah wrote:
>>>>>>>>> *"suppress" is the opposite of "allow", so
>>>>>>>>> setting suppress.auto.create.topics=false would mean
that we do
>>>>>> _not_
>>>>>>>> allow
>>>>>>>>> auto topic creation; when set to true, the server configuration
>>>> will
>>>>>>>>> determine whether we allow automatic creation or not.*
>>>>>>>>>
>>>>>>>>> Sorry, I meant suppress.auto.create.topics=true above
to
>> disallow
>>>>>> auto
>>>>>>>>> topic creation.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Aug 22, 2018 at 10:34 PM Dhruvil Shah <
>>>> dhruvil@confluent.io
>>>>>>>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> To be clear, we will allow auto topic creation only
when
>> server
>>>>>> config
>>>>>>>>>> auto.create.topics.enable=true and consumer config
>>>>>>>>>> allow.auto.create.topics=true; when either is false,
we would
>>>> not
>>>>>>> create
>>>>>>>>>> the topic if it does not exist.
>>>>>>>>>>
>>>>>>>>>> "suppress" is the opposite of "allow", so
>>>>>>>>>> setting suppress.auto.create.topics=false would mean
that we
>> do
>>>>>> _not_
>>>>>>>> allow
>>>>>>>>>> auto topic creation; when set to true, the server
>> configuration
>>>>>> will
>>>>>>>>>> determine whether we allow automatic creation or
not.
>>>>>>>>>>
>>>>>>>>>> I think "allow" is easier to understand but I am
open to
>>>>>> suggestions.
>>>>>>>>>>
>>>>>>>>>> - Dhruvil
>>>>>>>>>>
>>>>>>>>>> On Wed, Aug 22, 2018 at 6:53 PM Brandon Kirchner
<
>>>>>>>>>> brandon.kirchner@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> “allow=false” seems a bit more intuitive
to me than
>>>>>> “suppress=false”
>>>>>>>>>>>
>>>>>>>>>>> Brandon
>>>>>>>>>>>
>>>>>>>>>>>> On Aug 22, 2018, at 8:48 PM, Ted Yu <yuzhihong@gmail.com>
>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> We may also consider :
>>>>>>>>>>>>
>>>>>>>>>>>> "suppress.auto.topic.creation"
>>>>>>>>>>>>
>>>>>>>>>>>> or
>>>>>>>>>>>>
>>>>>>>>>>>> "allow.auto.topic.creation"
>>>>>>>>>>>>
>>>>>>>>>>>> w.r.t. suppress or allow, I don't have strong
opinion
>> either.
>>>>>> It's
>>>>>>>> just
>>>>>>>>>>> a
>>>>>>>>>>>> matter of choosing the proper default value.
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers
>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Aug 22, 2018 at 6:00 PM Dhruvil
Shah <
>>>>>> dhruvil@confluent.io
>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Matthias,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Do you mean something like "suppress.auto.create.topic"?
I
>> am
>>>>>>> leaning
>>>>>>>>>>> a bit
>>>>>>>>>>>>> towards "allow.auto.create.topics" but
I don't have a
>> strong
>>>>>>>> preference
>>>>>>>>>>>>> either. Let's wait to hear if anyone
else has an opinion on
>>>>>> this.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Dhruvil
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Aug 21, 2018 at 5:28 PM Matthias
J. Sax <
>>>>>>>> matthias@confluent.io
>>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for the KIP Dhruvil!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I agree with Jason's comment. An
alternative might be to
>> use
>>>>>>>>>>> "suppress"
>>>>>>>>>>>>>> what would revert the logic of "allow".
Not sure which one
>>>> is
>>>>>> more
>>>>>>>>>>>>>> intuitive and I am fine with both
(no personal
>> preference).
>>>>>> Just
>>>>>>>>>>> wanted
>>>>>>>>>>>>>> to mention it as an alternative.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Don't have any further comments/question
so far.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 8/21/18 4:42 PM, Jason Gustafson
wrote:
>>>>>>>>>>>>>>> Hey Dhruvil,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I would suggest using the verb
"allow" rather than
>> "enable.
>>>>>> The
>>>>>>>>>>>>> consumer
>>>>>>>>>>>>>>> cannot enable auto topic creation
because it is
>> configured
>>>> on
>>>>>> the
>>>>>>>>>>>>> broker.
>>>>>>>>>>>>>>> All it can do is prevent it from
happening if it is
>>>> enabled.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -Jason
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Aug 21, 2018 at 3:56
PM, Dhruvil Shah <
>>>>>>>> dhruvil@confluent.io>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I would like to start discussion
on KIP-361 that
>> proposes
>>>> we
>>>>>>> add a
>>>>>>>>>>>>>> consumer
>>>>>>>>>>>>>>>> configuration to disable
auto topic creation.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Link to the KIP:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+
>>>>>>>>>>>>>>>> Configuration+to+Disable+Auto+Topic+Creation
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Suggestions and feedback
are welcome!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Dhruvil
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
> 


Mime
View raw message