From dev-return-95256-archive-asf-public=cust-asf.ponee.io@kafka.apache.org Tue Jun 19 06:00:02 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id F2DA0180663 for ; Tue, 19 Jun 2018 06:00:01 +0200 (CEST) Received: (qmail 66783 invoked by uid 500); 19 Jun 2018 04:00:00 -0000 Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kafka.apache.org Delivered-To: mailing list dev@kafka.apache.org Received: (qmail 66739 invoked by uid 99); 19 Jun 2018 03:59:59 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Jun 2018 03:59:59 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 5622F1A0799 for ; Tue, 19 Jun 2018 03:59:59 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.969 X-Spam-Level: * X-Spam-Status: No, score=1.969 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=simplemachines-com-au.20150623.gappssmtp.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 2IQoNRHjaQh3 for ; Tue, 19 Jun 2018 03:59:57 +0000 (UTC) Received: from mail-vk0-f41.google.com (mail-vk0-f41.google.com [209.85.213.41]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id D30205F295 for ; Tue, 19 Jun 2018 03:59:56 +0000 (UTC) Received: by mail-vk0-f41.google.com with SMTP id 128-v6so10854929vkf.8 for ; Mon, 18 Jun 2018 20:59:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=simplemachines-com-au.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=1c+keRZcHKoVQ2e4Yx0n/UHp2jBzbHpZDtdgoYzTBQA=; b=t/Jq7Bcy2tWljLKmeq3cmYa6VB6ffuQeRR38wsS8lwlh0ShAGxi9jjLNNdyUIbi6lF gTAz9PW1yQaA1CmceOJIlMJUzUbU0Yut38kX92y0pfrkNuKF4BPR6tpyCfkQ/X6H0QTU 8Sg+MMNO72hcJ9dVNvMxZKYurdA9in/F+m++kPphEaBIE23pkrE39P2t23rHB/bZaMF6 Q6YdJ9qG2ncSfb7TgYQRa7HQpVXVLXCiYtW1M1j6cSTkhdfTMUv8v7S+bFI/GmFpM2tJ g36EIr9Uva63TusF/LSIf/LMlG7MyThON0piwtokuez8jHlO/5sODgCibyYhji5spq3V 5Utw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=1c+keRZcHKoVQ2e4Yx0n/UHp2jBzbHpZDtdgoYzTBQA=; b=DzF4zFIh3LxeBtDDMn/MX0A8OaXi69NGUjxWtT6lHcnH9fwTwSc9wrsN3rWZ/0ViHP jsx640lWqZZ50FYRdYfj54yralSkh5eDNyXV6R3DngeayCQsM9r6kPjFGgC9o4sXIXrA OwqGlu54MN/hMKijnS6Z43SfDRJCE7ne+6TnwpUdLI5I8WNHOT47YBiKtL8G8Wp4okYL mcOaPlibJBvHovV9A/ht/TMTUK9bnXWrCetMThrIwc5GSiCpCUSy9cyuHFHoQ2kiNjTS MrpsFKt/eUdH9xKLQZg2FxvtORdfuGqEBd09q7F09BvsX7knrQiRMusq7s/QvU8vlvvL Na7g== X-Gm-Message-State: APt69E2e4G4OZ/G1mpX3tZmxgCIQzgirD75vaY8d7dqSeu95jAsDiwH1 ndV9HldVBS3otAZG3owDHeSrHlXK+8eTNeOHhqTx81Wj X-Google-Smtp-Source: ADUXVKJ9mZVbl5Tn6lsr6lo9cN+BWtez4CMqB34IcaZL24CK2JNiznNQBmKVR2f+aZCVn63/WPUfFW/0DCCOENvzdJg= X-Received: by 2002:a1f:5fc1:: with SMTP id t184-v6mr8584776vkb.140.1529380789713; Mon, 18 Jun 2018 20:59:49 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a67:7ac5:0:0:0:0:0 with HTTP; Mon, 18 Jun 2018 20:59:48 -0700 (PDT) In-Reply-To: References: From: Stephane Maarek Date: Tue, 19 Jun 2018 11:59:48 +0800 Message-ID: Subject: Re: [DISCUSS] KIP-317: Transparent Data Encryption To: dev@kafka.apache.org Content-Type: multipart/alternative; boundary="000000000000b53a7c056ef6b663" --000000000000b53a7c056ef6b663 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Sonke Very much needed feature and discussion. FYI the image links seem broken. My 2 cents (if I understood correctly): you say "This process will be implemented after Serializer and Interceptors are done with the message right before it is added to the batch to be sent, in order to ensure that existing serializers and interceptors keep working with encryption just like without it." I think encryption should happen AFTER a batch is created, right before it is sent. Reason is that if we want to still keep advantage of compression, encryption needs to happen after it (and I believe compression happens on a batch level). So to me for a producer: serializer / interceptors =3D> batching =3D> compression =3D> encryption =3D> send. and the inverse for a consumer. Regards Stephane On 19 June 2018 at 06:46, S=C3=B6nke Liebau wrote: > Hi everybody, > > I've created a draft version of KIP-317 which describes the addition > of transparent data encryption functionality to Kafka. > > Please consider this as a basis for discussion - I am aware that this > is not at a level of detail sufficient for implementation, but I > wanted to get some feedback from the community on the general idea > before spending more time on this. > > Link to the KIP is: > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 317%3A+Add+transparent+data+encryption+functionality > > Best regards, > S=C3=B6nke > --000000000000b53a7c056ef6b663--