Mailing-List: contact user-help@kudu.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@kudu.apache.org
Date: Tue, 25 Apr 2017 02:08:55 +0000 (UTC)
From: fventuri@comcast.net
To: user@kudu.apache.org
Message-ID: <1668036986.13703141.1493086135916.JavaMail.zimbra@comcast.net>
In-Reply-To: <793726984.13692432.1493084898985.JavaMail.zimbra@comcast.net>
Subject: Data encryption in Kudu
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_13703140_1414164905.1493086135915"
Thread-Topic: Data encryption in Kudu
Thread-Index: K7r7y7DGsTvCS+4YdolYMJ4vB1q/zQ==
archived-at: Tue, 25 Apr 2017 02:09:02 -0000

------=_Part_13703140_1414164905.1493086135915
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Over the weekend I started looking at what it would take to add data encryption to Kudu (besides using filesystem encryption via dm-crypt or something like that). 

Here are a few notes - please feel free to comment on them and add suggestions: 

- reading through this mailing list, it looks like this feature has been asked a couple of times but last year, but from what I can tell, noone is currently working on it. 
- a client-based approach to encryption like the one used by HDFS wouldn't work (at least out of the box) because for instance encrypting the primary key at the client would prevent being able to have range filters for scans; it might work for the columns that are not part of the primary key 
- there's already code in Kudu for several compression codecs (LZ4, gzip, etc); I thought it would be possible to add similar code for encryption codecs (to be applied after the compression, of course) 
- the WAL log files and delta files should be similarly encrypted too 
- not sure what would be the best way to manage the key - I see that in HDFS they use a double key mechanism, where the encryption key for the data file is itself encrypted with the allowed user key and this whole process is managed by an external Key Management Service 

Thanks in advance for your ideas and suggestions, 
Franco 

------=_Part_13703140_1414164905.1493086135915
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable

<html><body><div style=3D"font-family: Arial; font-size: 12pt; color: #0000=
00"><div>Over the weekend I started looking at what it would take to add da=
ta encryption to Kudu (besides using filesystem encryption via dm-crypt or =
something like that).</div><div><br></div><div>Here are a few notes - pleas=
e feel free to comment on them and add suggestions:</div><div><br></div><di=
v>- reading through this mailing list, it looks like this feature has been =
asked a couple of times but last year, but from what I can tell, noone is c=
urrently working on it.</div><div>- a client-based approach to encryption l=
ike the one used by HDFS wouldn't work (at least out of the box) because fo=
r instance encrypting the primary key at the client would prevent being abl=
e to have range filters for scans; it might work for the columns that are n=
ot part of the primary key</div><div>- there's already code in Kudu for sev=
eral compression codecs (LZ4, gzip, etc); I thought it would be possible to=
 add similar code for encryption codecs (to be applied after the compressio=
n, of course)</div><div>- the WAL log files and delta files should be simil=
arly encrypted too</div><div>- not sure what would be the best way to manag=
e the key - I see that in HDFS they use a double key mechanism, where the e=
ncryption key for the data file is itself encrypted with the allowed user k=
ey and this whole process is managed by an external Key Management Service<=
/div><div><br></div><div>Thanks in advance for your ideas and suggestions,<=
/div><div>Franco&nbsp;</div></div></body></html>
------=_Part_13703140_1414164905.1493086135915--