From dev-return-106325-archive-asf-public=cust-asf.ponee.io@kafka.apache.org Tue Aug 6 14:42:02 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id ECAC1180595 for ; Tue, 6 Aug 2019 16:42:01 +0200 (CEST) Received: (qmail 45946 invoked by uid 500); 6 Aug 2019 14:41:56 -0000 Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kafka.apache.org Delivered-To: mailing list dev@kafka.apache.org Received: (qmail 45935 invoked by uid 99); 6 Aug 2019 14:41:56 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Aug 2019 14:41:56 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id D60C51A42C7 for ; Tue, 6 Aug 2019 14:41:55 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3 X-Spam-Level: *** X-Spam-Status: No, score=3 tagged_above=-999 required=6.31 tests=[HEADER_FROM_DIFFERENT_DOMAINS=0.249, HTML_MESSAGE=2, KAM_INFOUSMEBIZ=0.75, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id ByGG-IaFq9Ak for ; Tue, 6 Aug 2019 14:41:51 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.221.43; helo=mail-wr1-f43.google.com; envelope-from=ismaelj@gmail.com; receiver= Received: from mail-wr1-f43.google.com (mail-wr1-f43.google.com [209.85.221.43]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id 49D5C7DD30 for ; Tue, 6 Aug 2019 14:41:51 +0000 (UTC) Received: by mail-wr1-f43.google.com with SMTP id n9so88293974wru.0 for ; Tue, 06 Aug 2019 07:41:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=7+KndoapCsyuuse/eCvb4gIKziTXJMSaghRXypU5YKI=; b=Q2qamoaf+3MOevQ+jtZFLsRcFq0DNpfrmC3X6a2g4YfFJShQ0+PqERSZuNu4mDtagA XLvZQeZHWmeewvZwsg193hrijqiyovsI+0hE9M2Zh9rNzLW52ZtNZY3jan+EE891L+xc vQo+jIZSmnfVXcTh07gFKBGdoty+K/bT0NE6+3tGzIdMUKKnHLD04KhZcWE47VkAVyeb odcZy4ZxVtIMd1ecPL5hZQTQySGOVN5EE9QMSt61TO32cEv9CLUbIXp01tG4KplrHRbA yNk40BqYkHdU3ubLLmPndRIEJVdOjpX8Xap7FjARFRRGdgk9ZGop65ZH5+6H/1uKkPCe g0Lw== X-Gm-Message-State: APjAAAUfAAmuSX+wTRp+T9oiOIZArsnSMbCBXRMzMPkv+WG9Awq/8gtO 4HvwYN5O7j5pocJUHAoCDwSxjPSoy4YTG4BuAI2qRQ== X-Google-Smtp-Source: APXvYqxTz3ZpbOtexD9cDDOYCuxwCMmRuYjcZVepn79/SjUEbOEef2KQGIoydK+p4c0gielpEJyExAdMAn+qEr8nswQ= X-Received: by 2002:adf:fe09:: with SMTP id n9mr5489948wrr.41.1565102505108; Tue, 06 Aug 2019 07:41:45 -0700 (PDT) MIME-Version: 1.0 References: <6028cb4f-596b-4f67-88fb-e31981766e67@www.fastmail.com> <39a4650e-2aaf-4beb-a2b5-4b284637318d@www.fastmail.com> In-Reply-To: From: Ismael Juma Date: Tue, 6 Aug 2019 07:41:08 -0700 Message-ID: Subject: Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer To: dev Content-Type: multipart/alternative; boundary="000000000000daa8b9058f73d27f" --000000000000daa8b9058f73d27f Content-Type: text/plain; charset="UTF-8" Hi Harsha, I mentioned policies and the authorizer. For example, with CreateTopicPolicy, you can implement the limits you describe. If you have ideas of how that should be improved, please submit a KIP. My point is that this KIP is not introducing any new functionality with regards to what rogue clients can do. It's using the existing protocol that is already exposed via the AdminClient. So, I don't think we need to address it in this KIP. Does that make sense? Ismael On Tue, Aug 6, 2019 at 7:13 AM Harsha Chintalapani wrote: > Ismael, > Sure AdminClient can do that and we should've shipped a config or > use the existing one to block that. Not all users are yet to upgrade to > AdminClient and start using that to cause issues yet. > In shared environment we should allow server to set sane > defaults and not allow every client to go ahead create random no.of > topic/partitions and replication factor. Even if the users want to allow > topic creation proposed in the KIP , it makes sense to have some guards > against the no.of partitions and replication factor. Authorizer is not > always an answer to block requests and having users set server side configs > to protect a multi-tenant environment is required. In a non-secure > environment Authorizer is a blunt instrument either you end up blocking > everyone or allowing everyone. > I am asking to have server side that allow clients to create topics or not > , if they are allowed set a ceiling on max no.of partitions and > replication-factor. > > -Harsha > > > > On Mon, Aug 5 2019 at 8:58 PM, wrote: > > > Harsha, > > > > Rogue clients can use the admin client to create topics and partitions. > > ACLs and policies can help in that case as well as this one. > > > > Ismael > > > > On Mon, Aug 5, 2019, 7:48 PM Harsha Chintalapani > wrote: > > > > Hi Justine, > > Thanks for the KIP. > > "When server-side auto-creation is disabled, client-side auto-creation > > will try to use client-side configurations" > > If I understand correctly, this KIP is removing any server-side blocking > > client auto creation of topic? > > if so this will present potential issue of rogue client creating ton of > > topic-partitions and potentially bringing down the service for everyone > or > > degrade the service itself. > > By reading the KIP its not clear to me that there is a clear way to block > > auto creation topics of all together from clients by server side config. > > Server side configs of default topic, partitions should take higher > > precedence and client shouldn't be able to create a topic with higher > no.of > > partitions, replication than what server config specifies. > > > > > > > > > Thanks, > > Harsha > > > > > > > > > > > > > > > On Mon, Aug 05, 2019 at 5:24 PM, Justine Olshan > > wrote: > > > > > > > > > > Hi all, > > > I made some changes to the KIP. Hopefully this configuration change > will > > > make things a little clearer. > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/ > > > KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer > > > > > > Please let me know if you have any feedback or questions! > > > > > > Thank you, > > > Justine > > > > > > On Wed, Jul 31, 2019 at 1:44 PM Colin McCabe > > wrote: > > > > > > Hi Mickael, > > > > > > I think you bring up a good point. It would be better if we didn't ever > > > have to set up client-side configuration for this feature, and KIP-464 > > > would let us skip this entirely. > > > > > > On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote: > > > > > > Hi Mickael, > > > I agree that KIP-464 works on newer brokers, but I was a bit worried > how > > > things would play out on older brokers that* do not *have KIP 464 > > > > > > included. > > > > > > Is it enough to throw an error in this case when producer configs are > > not > > > specified? > > > > > > I think the right thing to do would be to log an error message in the > > > client. We will need to have that capability in any case, to cover > > > scenarios like the client trying to auto-create a topic that they don't > > > have permission to create. Or a client trying to create a topic on a > > broker > > > so old that CreateTopicsRequest is not supported. > > > > > > The big downside to relying on KIP-464 is that it is a very recent > > feature > > > -- so recent that it hasn't even made its way to any official Apache > > > release. It's scheduled for the upcoming 2.4 release in a few months. > > > > > > So if you view this KIP as a step towards removing broker-side > > > auto-create, you might want to support older brokers just to accelerate > > > adoption, and hasten the day when we can finally flip broker-side > > > auto-create to off (or even remove it entirely). > > > > > > I have to agree, though, that having client-side configurations for > > number > > > of partitions and replication factor is messy. Maybe it would be worth > > it > > > to restrict support to post-KIP-464 brokers, if we could avoid creating > > > more configs. > > > > > > best, > > > Colin > > > > > > On Wed, Jul 31, 2019 at 9:10 AM Mickael Maison < > mickael.maison@gmail.com > > > > > > wrote: > > > > > > Hi Justine, > > > > > > We can rely on KIP-464 which allows to omit the partition count or > > > replication factor when creating a topic. In that case, the broker > > defaults > > > are used. > > > > > > On Wed, Jul 31, 2019 at 4:55 PM Justine Olshan > > > wrote: > > > > > > Michael, > > > That makes sense to me! > > > To clarify, in the current state of the KIP, the producer does not > > > > > > rely > > > > > > on > > > > > > the broker to autocreate--if the broker's config is disabled, then > > > > > > the > > > > > > producer can autocreate on its own with a create topic request (the > > > > > > same > > > > > > type of request the admin client uses). > > > However, if both configs are enabled, the broker will autocreate > > > > > > through > > > > > > a > > > > > > metadata request before the producer gets a chance. Of course, the way > > to > > > avoid this, is to do as you suggested, and set > > > > > > the > > > > > > "allow_auto_topic_creation" field to false. > > > > > > I think the only thing we need to be careful with in this setup is > > > > > > without > > > > > > KIP 464, we can not use broker defaults for this topic. A user needs > > > > > > to > > > > > > specify the number of partition and replication factor in the config. > An > > > alternative to this is to have coded defaults for when these > > > > > > configs > > > > > > are > > > > > > unspecified, but it is not immediately apparent what these defaults > > > > > > should > > > > > > be. > > > > > > Thanks again for reading my KIP, > > > Justine > > > > > > On Wed, Jul 31, 2019 at 4:19 AM Mickael Maison < > > > > > > mickael.maison@gmail.com > > > > > > wrote: > > > > > > Hi Justine, > > > > > > Thanks for the response! > > > In my opinion, it would be better if the producer did not rely at > > > > > > all > > > > > > on the broker auto create feature as this is what we're aiming to > > > deprecate. When requesting metadata we can set the > > > "allow_auto_topic_creation" field to false to avoid the broker auto > > > creation. Then if the topic is not existing, send a CreateTopicRequest. > > > > > > What do you think? > > > > > > On Mon, Jul 29, 2019 at 6:34 PM Justine Olshan < > > > > > > jolshan@confluent.io> > > > > > > wrote: > > > > > > Currently the way it is implemented, the broker auto-creation > > > > > > configuration > > > > > > takes precedence. The producer will not use the CreateTopics > > > > > > request. > > > > > > (Technically it can--but the topic will already be created > > > > > > through > > > > > > the > > > > > > broker, so it will never try to create the topic.) It is possible to > > > change this however, and I'd be happy to > > > > > > discuss > > > > > > the > > > > > > benefits of this alternative. > > > > > > Thank you, > > > Justine > > > > > > On Mon, Jul 29, 2019 at 10:26 AM Mickael Maison < > > > > > > mickael.maison@gmail.com> > > > > > > wrote: > > > > > > Hi Justine, > > > > > > Thanks for the KIP! > > > > > > In case auto creation is enabled on both the client and server, > > > > > > will > > > > > > the producer still use the AdminClient (CreateTopics request) > > > > > > to > > > > > > create topics? and not rely on the broker auto create. I'm guessing the > > > answer is yes but can you make it explicit. > > > > > > Thank you > > > > > > On Wed, Jul 24, 2019 at 6:23 PM Justine Olshan < > > > > > > jolshan@confluent.io> > > > > > > wrote: > > > > > > Hi, > > > Just a friendly reminder to take a look at this KIP if you > > > > > > have > > > > > > the > > > > > > time. > > > > > > I was thinking about broker vs. client default precedence, > > > > > > and I > > > > > > think it > > > > > > makes sense to keep the broker as the default used when both > > > > > > client-side > > > > > > and broker-side defaults are configured. The idea is that > > > > > > there > > > > > > would be > > > > > > pretty clear documentation, and that many systems with > > > > > > configurations > > > > > > that > > > > > > the client could not change would likely have the auto-create > > > > > > default > > > > > > off. > > > > > > (In cloud for example). > > > > > > It also seems like in most cases, the consumer config > > > 'allow.auto.create.topics' was created to actually prevent > > > > > > the > > > > > > creation > > > > > > of > > > > > > topics, so the loss of creation functionality will not be a > > > > > > big > > > > > > problem. > > > > > > I'm happy to discuss any other compatibility problems or > > > > > > components > > > > > > of > > > > > > this KIP. > > > > > > Thank you, > > > Justine > > > > > > On Wed, Jul 17, 2019 at 9:11 AM Justine Olshan < > > > > > > jolshan@confluent.io > > > > > > wrote: > > > > > > Hello all, > > > > > > I was looking at this KIP again, and there is a decision I > > > > > > made > > > > > > that I > > > > > > think is worth discussing. > > > > > > In the case where both the broker and producer's > > > 'auto.create.topics.enable' are set to true, we have to > > > > > > choose > > > > > > either > > > > > > the > > > > > > broker configs or the producer configs for the replication > > > factor/partitions. > > > > > > Currently, the decision is to have the broker defaults take > > > > > > precedence. > > > > > > (It is easier to do this in the implementation.) It also > > > > > > makes > > > > > > some > > > > > > sense > > > > > > for this behavior to take precedence since this behavior > > > > > > already > > > > > > occurs as > > > > > > the default. > > > > > > However, I was wondering if it would be odd for those who > > > > > > can > > > > > > only > > > > > > see > > > > > > the > > > > > > producer side to set configs for replication > > > > > > factor/partitions > > > > > > and > > > > > > see > > > > > > different results. Currently the documentation for the > > > > > > config > > > > > > states > > > > > > that > > > > > > the config values are only used when the broker config is > > > > > > not > > > > > > enabled, > > > > > > but > > > > > > this might not always be clear to the user. Changing the > > > > > > code > > > > > > to > > > > > > have > > > > > > the > > > > > > producer's configurations take precedence is possible, but > > > > > > I > > > > > > was > > > > > > wondering > > > > > > what everyone thought. > > > > > > Thank you, > > > Justine > > > > > > On Fri, Jul 12, 2019 at 2:49 PM Justine Olshan < > > > > > > jolshan@confluent.io> > > > > > > wrote: > > > > > > Just a quick update-- > > > > > > It seems that enabling both the broker and producer > > > > > > configs > > > > > > works > > > > > > fine, > > > > > > except that the broker configurations for partitions, > > > > > > replication > > > > > > factor > > > > > > take precedence. > > > I don't know if that is something we would want to > > > > > > change, but > > > > > > I'll be > > > > > > updating the KIP for now to reflect this. Perhaps we would > > > > > > want to > > > > > > add more > > > > > > to the documentation of the the producer configs to > > > > > > clarify. > > > > > > Thank you, > > > Justine > > > > > > On Fri, Jul 12, 2019 at 9:28 AM Justine Olshan < > > > > > > jolshan@confluent.io> > > > > > > wrote: > > > > > > Hi Colin, > > > > > > Thanks for looking at the KIP. I can definitely add to > > > > > > the > > > > > > title > > > > > > to > > > > > > make > > > > > > it more clear. > > > > > > It makes sense that both configurations could be turned > > > > > > on > > > > > > since > > > > > > there > > > > > > are many cases where the user can not control the > > > > > > server-side > > > > > > configurations. I was a little concerned about how both > > > > > > interacting > > > > > > would > > > > > > work out -- if there would be to many requests for new > > > > > > topics, > > > > > > for > > > > > > example. > > > > > > But it since it does make sense to allow both > > > > > > configurations > > > > > > enabled, I > > > > > > will test out some scenarios and I'll change the KIP to > > > > > > support > > > > > > this. > > > > > > I also agree with documentation about distinguishing the > > > > > > differences. I > > > > > > was having some trouble with the wording but I like the > > > > > > phrases > > > > > > "server-side" and "client-side." That's a good > > > > > > distinction I > > > > > > can > > > > > > use > > > > > > when > > > > > > describing. > > > > > > I'll try to update the KIP soon keeping everyone's input > > > > > > in > > > > > > mind. > > > > > > Thanks, > > > Justine > > > > > > On Thu, Jul 11, 2019 at 5:39 PM Colin McCabe < > > > > > > cmccabe@apache.org > > > > > > wrote: > > > > > > Hi Justine, > > > > > > Thanks for the KIP. This seems like a good step towards > > > > > > removing > > > > > > server-side topic auto-creation. > > > > > > We should add included "client-side" to the title of > > > > > > the KIP > > > > > > somewhere, > > > > > > to make it clear that we're talking about client-side > > > > > > auto > > > > > > creation. > > > > > > The KIP says: > > > > > > In order to automatically create topics with the > > > > > > producer, the > > > > > > producer's > > > > > > auto.create.topics.enable config must be set to true > > > > > > and > > > > > > the > > > > > > broker > > > > > > config should be set to false > > > > > > From a user's point of view, this seems > > > > > > counter-intuitive. > > > > > > In > > > > > > order to > > > > > > auto-create topics the broker's > > > > > > auto.create.topics.enable > > > > > > config > > > > > > should be > > > > > > set to false? It seems like the server-side > > > > > > auto-create is > > > > > > unrelated to > > > > > > the client-side auto-create. We could have both turned > > > > > > on > > > > > > (and > > > > > > I'm > > > > > > sure > > > > > > that in the real world, people will try this > > > > > > configuration...) > > > > > > There's no > > > > > > reason not to support this, I think. > > > > > > We should add some documentation explaining the > > > > > > difference > > > > > > between > > > > > > server-side and client-side auto-creation. Without > > > > > > documentation, > > > > > > an admin > > > > > > might think that they had disabled all forms of > > > > > > auto-creation by > > > > > > setting > > > > > > the -side setting to false-- but this is not the case, > > > > > > of > > > > > > course. > > > > > > best, > > > Colin > > > > > > On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote: > > > > > > Hi Dhruvil, > > > > > > Thanks for reading the KIP! > > > That was the general idea for deprecation. We would > > > > > > log a > > > > > > warning > > > > > > when the > > > > > > config is enabled on the broker. > > > I also believe that there would be a change to > > > > > > documentation. > > > > > > If there is anything else that should be done, please > > > > > > let > > > > > > me > > > > > > know! > > > > > > Justine > > > > > > On Thu, Jul 11, 2019 at 4:17 PM Dhruvil Shah < > > > > > > dhruvil@confluent.io> > > > > > > wrote: > > > > > > Hi Justine, > > > > > > Thanks for the KIP, this is great! > > > > > > Could you add some more information about what > > > > > > deprecating > > > > > > the > > > > > > broker > > > > > > configuration means? Would we log a warning in the > > > > > > logs > > > > > > when > > > > > > auto > > > > > > topic > > > > > > creation is enabled on the broker, for example? > > > > > > Thanks, > > > Dhruvil > > > > > > On Thu, Jul 11, 2019 at 10:28 AM Justine Olshan < > > > > > > jolshan@confluent.io> > > > > > > wrote: > > > > > > Hello all, > > > > > > I'd like to start a discussion thread for KIP-487. This KIP plans to > > > deprecate the current system of > > > > > > auto-creating > > > > > > topics > > > > > > through requests to the metadata and give the > > > > > > producer the > > > > > > ability to > > > > > > automatically create topics instead. > > > > > > More information can be found here: > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/ > > > KIP-487%3A+Automatic+Topic+Creation+on+Producer > > > > > > Thank you, > > > Justine Olshan > > > > > > > > > > > > > > --000000000000daa8b9058f73d27f--