From dev-return-91208-archive-asf-public=cust-asf.ponee.io@kafka.apache.org Wed Jan 24 22:00:08 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 42BD6180630 for ; Wed, 24 Jan 2018 22:00:08 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 332EB160C4E; Wed, 24 Jan 2018 21:00:08 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 78125160C39 for ; Wed, 24 Jan 2018 22:00:07 +0100 (CET) Received: (qmail 26046 invoked by uid 500); 24 Jan 2018 21:00:06 -0000 Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kafka.apache.org Delivered-To: mailing list dev@kafka.apache.org Received: (qmail 25890 invoked by uid 99); 24 Jan 2018 21:00:05 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Jan 2018 21:00:05 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id DBC86180701 for ; Wed, 24 Jan 2018 21:00:02 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.121 X-Spam-Level: X-Spam-Status: No, score=-0.121 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=opencore.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id ovNLpaiCTzur for ; Wed, 24 Jan 2018 21:00:00 +0000 (UTC) Received: from mail-yb0-f178.google.com (mail-yb0-f178.google.com [209.85.213.178]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id D87165F1B3 for ; Wed, 24 Jan 2018 20:59:59 +0000 (UTC) Received: by mail-yb0-f178.google.com with SMTP id v12so2096104ybl.3 for ; Wed, 24 Jan 2018 12:59:59 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to :content-transfer-encoding; bh=NKptYLfJxF+zTQ7DqUVu3S58fuTyra3iZ2HY2x3zvjU=; b=kH8pFQxFInki9Ylni7yUI1aKBSsZK+BBKBX9UUcpNXHEdRijv9YjQrQ549fzCWHc6c f749EM6vXqv3StyE3lmkDffvNOod7Z1UWVu8Qc84yszgj8EnfvfZtJ7LyzaQZoE+f0B2 j0yd7ZF2yxQdv9mhUEuVKVWV5lz5BQhcG/sYku6W0XvgJ63HSJ38l89lPOJ9kNZ9uy+N o6jhlwdkGeTW3cpx4//RGM7jmXM/sDWsme5hqqlHd/K/ffutgcwpNMhBm8E3IEY3taub IGMoLuBaYCNlwWCkT2jR5I7co3iCRWrmYwSkCq/Y+TiJFhgO/5hpgvUJ4U2NGR24F5o8 a5Xg== X-Gm-Message-State: AKwxytfGjmhPrrNigOAXyCkgxS82GXfUCiaFSiDCUehgKA83TV7Yu8Ns xzP/qFEd5Caq1QPE6NWkV+NFWXSmUK/pqRsdT4xjxzvtwAo= X-Google-Smtp-Source: AH8x227M6TYi2Bf3OMyxngcwX41PyBHeThy0ZWdVayEhXl0ioMjwF0gVZyhG4p+bEGAbsSHOiHDmEwZEtkQC4yLqLow= X-Received: by 10.37.66.20 with SMTP id p20mr6690163yba.207.1516827598421; Wed, 24 Jan 2018 12:59:58 -0800 (PST) MIME-Version: 1.0 Received: by 10.129.153.68 with HTTP; Wed, 24 Jan 2018 12:59:17 -0800 (PST) From: =?UTF-8?Q?S=C3=B6nke_Liebau?= Date: Wed, 24 Jan 2018 21:59:17 +0100 Message-ID: Subject: [DISCUSS] Improving ACLs by allowing ip ranges and subnet expressions? To: dev@kafka.apache.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi everyone, the current ACL functionality in Kafka is a bit limited concerning host based rules when specifying multiple hosts. A common scenario for this would be that if have a YARN cluster running Spark jobs that access Kafka and want to create ACLs based on the ip addresses of the cluster nodes. Currently kafka-acls only allows to specify individual ips, so this would look like ./kafka-acls --add --producer \ --topic test --authorizer-properties zookeeper.connect=3Dlocalhost:2181 \ --allow-principal User:spark \ --allow-host 10.0.0.10 \ --allow-host 10.0.0.11 \ --allow-host ... which can get unwieldy if you have a 200 node cluster. Internally this command would not create a single ACL with multiple host entries, but rather one ACL per host that is specified on the command line, which makes the ACL listing a bit confusing. There are currently a few jiras in various states around this topic: KAFKA-3531 [1], KAFKA-4759 [2], KAFKA-4985 [3] & KAFKA-5713 [4] KAFKA-4759 has a patch available, but would currently only add interpretation of CIDR notation, no specific ranges, which I think could easily be added. Colin McCabe commented in KAFKA-4985 that so far this was not implemented as no standard for expressing ip ranges with a fast implementation had been found so far, the available patch uses the ipmath [5] package for parsing expressions and range checking - which seems fairly small and focused. This would allow for expressions of the following type: 10.0.0.1 10.0.0.1-10.0.0.10 10.0.0.0/24 I'd suggest extending this a little to allow a semicolon separated list of values: 10.0.0.1;10.0.0.1-10.0.0.10;10.0.0.0/24 Performance considerations Internally the ipmath package represents ip addresses as longs, so if we stick with the example of a 200 node cluster from above, with the current implementation that would be 200 string comparisons for every request, whereas with a range it could potentially come down to two long comparisons. This is of course a back-of-the-envelope calculation at best, but there at least seems to be a case for investigating this a bit further I think. These changes would probably necessitate a KIP - though with some consideration they could be made in a way that no existing public facing functionality is changed, but for transparency and proper documentation I'd say a KIP would be preferable. I'd be happy to draft one if people think this is worthwhile. Let me know what you think. best regards, S=C3=B6nke [1] https://issues.apache.org/jira/browse/KAFKA-3531 [2] https://issues.apache.org/jira/browse/KAFKA-4759 [3] https://issues.apache.org/jira/browse/KAFKA-4985 [4] https://issues.apache.org/jira/browse/KAFKA-5713 [5] https://github.com/jgonian/commons-ip-math