From dev-return-37586-archive-asf-public=cust-asf.ponee.io@ignite.apache.org Thu Aug 9 14:48:42 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 1500A18065B for ; Thu, 9 Aug 2018 14:48:41 +0200 (CEST) Received: (qmail 27038 invoked by uid 500); 9 Aug 2018 12:48:41 -0000 Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ignite.apache.org Delivered-To: mailing list dev@ignite.apache.org Received: (qmail 27023 invoked by uid 99); 9 Aug 2018 12:48:40 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Aug 2018 12:48:40 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 11882C1BC1 for ; Thu, 9 Aug 2018 12:48:40 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.238 X-Spam-Level: X-Spam-Status: No, score=0.238 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.25, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id avWlLqnzPqFR for ; Thu, 9 Aug 2018 12:48:38 +0000 (UTC) Received: from mail-wr1-f67.google.com (mail-wr1-f67.google.com [209.85.221.67]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 5560E5F3EB for ; Thu, 9 Aug 2018 12:48:38 +0000 (UTC) Received: by mail-wr1-f67.google.com with SMTP id g6-v6so5068518wrp.0 for ; Thu, 09 Aug 2018 05:48:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:message-id:subject:to:date:in-reply-to:references :mime-version; bh=aVRMGGFvw1Y/5ERDe8GTO8ni2VLhaihaXUmjnaPKK/4=; b=RnZGSh1g9oHS1nDnx4d9ziMLtok3cCz282TXbAW2ghi6422dT31F6ocdHc9ZZwebnZ EbCGeaEKS2wiSvesQ2ilQ0gcOcHVp/L44SrX861KhbnNOPe6tVMfbo3CLmLZn3xJdzEB 6J4Q6rAWz17YaLzLxJIiidkujy0R/+qBFAcpaPdislFsDrLaF+GYH2KKy0Fmmsr6HZrL 58sXuScRsQeuy/lkyDowiMvppSWeCBvSNdEpfoUQ+aRXq4GyzjVnxKneMRXv4q9GG2kh tYxeArcIBFxgGSM+d7vCO/0Z0CWQZv4z8on8MVwq0vLfw5OpG1CB7qLrhnCTwEJ88KTR 8amA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:message-id:subject:to:date :in-reply-to:references:mime-version; bh=aVRMGGFvw1Y/5ERDe8GTO8ni2VLhaihaXUmjnaPKK/4=; b=gtGQ00LEEcP370GpAHA0b7cScIl4JKwKOpf52PrbQzkKpHI/HT5zeBMEvh21M6lE3O tBMV8Z15Yj0/ZMBKuJQ38jOGZJmWBlwtdx/FPcSgnOGrbslrwkcI5/fjVosMkJhaGva+ lcaMbnZDkeITn2q87ZtWc16qUBX7om254BAdyWGW3iy/yq/lJ/RDSyjlasYfjjYawJ00 UBBYrjEeJfoEqCKvZOEmA9Tec69AuK2K98JcnoHlWajX6TRRUx+O5Dfg49bB7ZnSH5sq Q9LZskKs5KqSFU1mOuGKvTB41OXi6H/mRQjUV0k+3EWiO9IFuD8EKoJ7EVupf2LBkeQC MNsQ== X-Gm-Message-State: AOUpUlFbqDiiECEBM9293WinGtO0ozyq66nSGkl72cZmhBIGBmI0aruZ SQwiObNrj0ZHnpfMakAEqypveKYk X-Google-Smtp-Source: AA+uWPxTkCNbat2dFRHL5UGw98oWcfUvj+rzqCpMEra/JuCd7YVsRDOUK73rivsG8otrn0TH1sXSqA== X-Received: by 2002:adf:9203:: with SMTP id 3-v6mr1295022wrj.275.1533818917791; Thu, 09 Aug 2018 05:48:37 -0700 (PDT) Received: from inca-pf0m1694.sigma.sbrf.ru ([194.186.207.245]) by smtp.googlemail.com with ESMTPSA id 139-v6sm15806138wmp.4.2018.08.09.05.48.36 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 09 Aug 2018 05:48:36 -0700 (PDT) Sender: =?UTF-8?B?0J3QuNC60L7Qu9Cw0Lkg0JjQttC40LrQvtCy?= From: Nikolay Izhikov X-Google-Original-From: Nikolay Izhikov Message-ID: Subject: TDE Implementation details. To: dev@ignite.apache.org Date: Thu, 09 Aug 2018 15:48:24 +0300 In-Reply-To: References: Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-nhxYs5SAFJxH7kaHTckl" X-Mailer: Evolution 3.28.1-2 Mime-Version: 1.0 --=-nhxYs5SAFJxH7kaHTckl Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello, Igniters. I want to share with you TDE implementation details. I think it can simplify review and acception of TDE implementation. This mail is copy of wiki page [1]. Please, share your thoughts. TDE is ready for review [2].=20 Please, let me know, who is able to make final review. This document describes some internal details of TDE.Phase 1 implementation= [3]. I suggest that reader familiar with the general design described in IEP-18 = [4].=20 Cache group key management and node join enhancements:=20 1. GridEncryptionManager encapsulates all logic related to key management:= =20 a. All group encryption keys are stored in MetaStore. b. Joining node sends to cluster: * Master key digest.=20 * All group keys saved locally (Map). Keys send over a = network in encrypted form. c. Coordinator on new node join: * Compares new node master key digest with the local master key digest.= =20 If differs then new node join is rejected. * Compares local and received group keys. If some key differs then new node join is rejected.=20 d. All server nodes: * If some of received keys are new then they save locally. 2. Dynamic cache creation: a. On server node - Encryption key is generated and added to DynamicCache= ChangeRequest. b. On client node: * Prior to creation of DynamicCacheChangeRequest we have to get new encr= yption key from some server node. * New request added to solve this - GenerateEncryptionKeyRequest. WAL Record encryption:=20 1. New type of WAL record created =E2=80=93 EncryptedRecord. 2. EncryptedRecord is a container that stores some WalRecordCacheGroupAwar= e in encrypted form inside. 3. Write: Each record for an encrypted group that implements WalRecordCacheGroupAwar= e written to WAL in encrypted form. Instead of that record we write EncryptedRecord with plain record inside(P= ageSnapshot, PageDeltaRecord, etc). 4. Read: There are 3 different cases on EncryptedRecord read: a. WAL restore =E2=80=93 we read EncryptedRecord, decrypt internal record= and return it. b. Offline WAL iteration(StandaloneWalRecordsIterator) - EncryptionSpi is= null so wecan=E2=80=99t decrypt EncryptedRecord and just return it from an= iterator. c. Meta storage restore process =E2=80=93 On node start we restore MetaSt= ore. When we do it no encryption keys are available, because they are stored i= n MetaStore. So we can=E2=80=99t decrypt EncryptedRecord and just return it from an it= erator. =20 We don't need decrypted record on this step to operate properly. Page encryption:=20 =09 1. We have to write on disk pages aligned on 2^n (2kb, 4kb, etc) for gain = maximum perfrormance.=20 =09 2. There is a 16 byte overhead for and AES CBC mode.=20 =09 3. So we have to preserve 16 bytes in page in memory to get encrypted page= size equal to 2^n when written it to disk.=20 =09 4. PageIO has many methods with pageSize parameter.=20 So for encrypted cache groups page size is calculated as cfg.getPageSize()= - 16 byte.=20 =09 [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=3D8906= 7473 [2] https://github.com/apache/ignite/pull/4167 [3] https://issues.apache.org/jira/browse/IGNITE-8485 [4] https://cwiki.apache.org/confluence/display/IGNITE/IEP-18%3A+Transparen= t+Data+Encryption=20 --=-nhxYs5SAFJxH7kaHTckl Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iQEzBAABCgAdFiEEOiTcLcdgyP2exB5ZbiaPbjg91GUFAltsOBgACgkQbiaPbjg9 1GXwbAf/Rku5QsHQeGQRURJVXbkPU9BEoN4UhEy6aFyd6ic0LESqo/1iW/LoDCBw s5ADDulWN8ijhU5eqG124L9D6x+6UEI40VwfHPGo+lkhfQuGYOe9mB3vfqOwJq23 c4jwR1Unh6S6lsnXd5BLaLwvG0/O8UVvonKoxxqs+f4zffpLbHaXi84NwKhxb73O nJcttAJhLqtLSb6A76fbEv635n4/cZa/5O1gfXNnwINWkGoU+fpN2pAN9dKQCsxS 8qHCF/srQYW8hnFx350gbDD517Ol0ontgvCqCaz3gF1s4Nwg0qTwRviSpK9lEP3g paELHrbBdf81KhMqOJSu3SnPFJ+F1w== =SUnL -----END PGP SIGNATURE----- --=-nhxYs5SAFJxH7kaHTckl--