hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Moist (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15006) Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS
Date Thu, 25 Jan 2018 20:50:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16339855#comment-16339855
] 

Steve Moist commented on HADOOP-15006:
--------------------------------------

I've attached a quick and dirty proof of concept. It's now using CryptoFSDataInput/OutputStream
to write encrypted data to S3.  It uses a constant key and iv.  I've mainly just ran the
entire unit/integration tests against it.  I get a bunch of failures as they compare data
byte for byte, and as expected they would fail.  But the rest of the suite passes with encryption
enabled.  Play around with it and let me know what you all think. 

> Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS
> ---------------------------------------------------------------
>
>                 Key: HADOOP-15006
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15006
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs/s3, kms
>            Reporter: Steve Moist
>            Priority: Minor
>         Attachments: S3-CSE Proposal.pdf, s3-cse-poc.patch
>
>
> This is for the proposal to introduce Client Side Encryption to S3 in such a way that
it can leverage HDFS transparent encryption, use the Hadoop KMS to manage keys, use the `hdfs
crypto` command line tools to manage encryption zones in the cloud, and enable distcp to copy
from HDFS to S3 (and vice-versa) with data still encrypted.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message