Mailing-List: contact user-help@kudu.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@kudu.apache.org
MIME-Version: 1.0
In-Reply-To: <CAAnRTpJMgNqB7L3543vkUmJZU353KpOVQ8Gw9+jnEmnhptQp7g@mail.gmail.com>
References: <793726984.13692432.1493084898985.JavaMail.zimbra@comcast.net>
 <1668036986.13703141.1493086135916.JavaMail.zimbra@comcast.net> <CAAnRTpJMgNqB7L3543vkUmJZU353KpOVQ8Gw9+jnEmnhptQp7g@mail.gmail.com>
From: Dan Burkert <danburkert@apache.org>
Date: Tue, 25 Apr 2017 10:38:46 -0700
Message-ID: <CALo2W-XesvT_Fr_zGbq-wE9FzpahVB9GqNGfVUS-qb6DWGpvBQ@mail.gmail.com>
Subject: Re: Data encryption in Kudu
To: user@kudu.apache.org
Content-Type: multipart/alternative; boundary=94eb2c0696888fee7d054e013463
archived-at: Tue, 25 Apr 2017 17:39:31 -0000

--94eb2c0696888fee7d054e013463
Content-Type: text/plain; charset=UTF-8

Hi Franco,

I think you are right that a client-based approach wouldn't work, because
we wouldn't want to encrypt at the level of individual cell values.  That
would get in the way of encoding, compression, predicate evaluation, etc.
As you note, adding encryption at the block layer is probably the way to
go.  Key management is definitely the tricky issue.  We do have one
advantage over HDFS - because Kudu does logical replication, the encryption
key can be scoped to a particular tablet server or tablet replica, it
wouldn't need to be shared among all replicas.  I haven't done enough
research to know if this makes it fundamentally easier to do key
management.  I would assume at a minimum we would want to integrate with
key providers such an HSM.  It would be good to have a thorough review of
existing solutions in the space, such as TDE
<https://en.wikipedia.org/wiki/Transparent_Data_Encryption> and the Hadoop
KMS.  Is this something you are interested in working on?

- Dan

On Tue, Apr 25, 2017 at 8:30 AM, David Alves <davidralves@gmail.com> wrote:

> Hi Franco
>
>   Dan, Alexey, Todd are our security experts.
>   Folks, thoughts on this?
>
> Best
> David
>
> On Mon, Apr 24, 2017 at 7:08 PM, <fventuri@comcast.net> wrote:
>
>> Over the weekend I started looking at what it would take to add data
>> encryption to Kudu (besides using filesystem encryption via dm-crypt or
>> something like that).
>>
>> Here are a few notes - please feel free to comment on them and add
>> suggestions:
>>
>> - reading through this mailing list, it looks like this feature has been
>> asked a couple of times but last year, but from what I can tell, noone is
>> currently working on it.
>> - a client-based approach to encryption like the one used by HDFS
>> wouldn't work (at least out of the box) because for instance encrypting the
>> primary key at the client would prevent being able to have range filters
>> for scans; it might work for the columns that are not part of the primary
>> key
>> - there's already code in Kudu for several compression codecs (LZ4, gzip,
>> etc); I thought it would be possible to add similar code for encryption
>> codecs (to be applied after the compression, of course)
>> - the WAL log files and delta files should be similarly encrypted too
>> - not sure what would be the best way to manage the key - I see that in
>> HDFS they use a double key mechanism, where the encryption key for the data
>> file is itself encrypted with the allowed user key and this whole process
>> is managed by an external Key Management Service
>>
>> Thanks in advance for your ideas and suggestions,
>> Franco
>>
>
>

--94eb2c0696888fee7d054e013463
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi Franco,<div><br></div><div>I think you are right that a=
 client-based approach wouldn&#39;t work, because we wouldn&#39;t want to e=
ncrypt at the level of individual cell values.=C2=A0 That would get in the =
way of encoding, compression, predicate evaluation, etc.=C2=A0 As you note,=
 adding encryption at the block layer is probably the way to go.=C2=A0 Key =
management is definitely the tricky issue.=C2=A0 We do have one advantage o=
ver HDFS - because Kudu does logical replication, the encryption key can be=
 scoped to a particular tablet server or tablet replica, it wouldn&#39;t ne=
ed to be shared among all replicas.=C2=A0 I haven&#39;t done enough researc=
h to know if this makes it fundamentally easier to do key management.=C2=A0=
 I would assume at a minimum we would want to integrate with key providers =
such an HSM.=C2=A0 It would be good to have a thorough review of existing s=
olutions in the space, such as <a href=3D"https://en.wikipedia.org/wiki/Tra=
nsparent_Data_Encryption" target=3D"_blank">TDE</a>=C2=A0and the Hadoop KMS=
.=C2=A0 Is this something you are interested in working on?</div><div><br><=
/div><div>- Dan</div><div class=3D"gmail_extra"><br><div class=3D"gmail_quo=
te">On Tue, Apr 25, 2017 at 8:30 AM, David Alves <span dir=3D"ltr">&lt;<a h=
ref=3D"mailto:davidralves@gmail.com" target=3D"_blank">davidralves@gmail.co=
m</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margi=
n:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr">=
Hi Franco<div><br></div><div>=C2=A0 Dan, Alexey, Todd are our security expe=
rts.</div><div>=C2=A0 Folks, thoughts on this?</div><div><br></div><div>Bes=
t</div><span class=3D"m_-4256221289814003803HOEnZb"><font color=3D"#888888"=
><div>David</div></font></span></div><div class=3D"m_-4256221289814003803HO=
EnZb"><div class=3D"m_-4256221289814003803h5"><div class=3D"gmail_extra"><b=
r><div class=3D"gmail_quote">On Mon, Apr 24, 2017 at 7:08 PM,  <span dir=3D=
"ltr">&lt;<a href=3D"mailto:fventuri@comcast.net" target=3D"_blank">fventur=
i@comcast.net</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" st=
yle=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>=
<div style=3D"font-family:Arial;font-size:12pt;color:#000000"><div>Over the=
 weekend I started looking at what it would take to add data encryption to =
Kudu (besides using filesystem encryption via dm-crypt or something like th=
at).</div><div><br></div><div>Here are a few notes - please feel free to co=
mment on them and add suggestions:</div><div><br></div><div>- reading throu=
gh this mailing list, it looks like this feature has been asked a couple of=
 times but last year, but from what I can tell, noone is currently working =
on it.</div><div>- a client-based approach to encryption like the one used =
by HDFS wouldn&#39;t work (at least out of the box) because for instance en=
crypting the primary key at the client would prevent being able to have ran=
ge filters for scans; it might work for the columns that are not part of th=
e primary key</div><div>- there&#39;s already code in Kudu for several comp=
ression codecs (LZ4, gzip, etc); I thought it would be possible to add simi=
lar code for encryption codecs (to be applied after the compression, of cou=
rse)</div><div>- the WAL log files and delta files should be similarly encr=
ypted too</div><div>- not sure what would be the best way to manage the key=
 - I see that in HDFS they use a double key mechanism, where the encryption=
 key for the data file is itself encrypted with the allowed user key and th=
is whole process is managed by an external Key Management Service</div><div=
><br></div><div>Thanks in advance for your ideas and suggestions,</div><div=
>Franco=C2=A0</div></div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div></div>

--94eb2c0696888fee7d054e013463--