hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Moist (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15006) Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS
Date Tue, 12 Dec 2017 22:39:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16288396#comment-16288396

Steve Moist commented on HADOOP-15006:

I don't think anyone's started it.  I posted the design doc in hopes of others looking at
it and critiquing it in the background while I focus on other things, that once enough people
had reviewed it, to start on it then.  The changes to the Hadoop CLI, KMS and other components
was what worried me about it.  It's bigger in scope than just S3a. 

In the proposal I made, we didn't have an issue with the cipher text length and plaintext
length as we used CTR with no padding vs the CBC with PKCS5Padding that the AWS sdk uses.
 I wrote a quick prototype using AES/CTR/NoPadding and ran all the integration tests against
it and it ran without issue and did diffs on the before/after of upload/download along with
TerraSort and had no issues.

> Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS
> ---------------------------------------------------------------
>                 Key: HADOOP-15006
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15006
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs/s3, kms
>            Reporter: Steve Moist
>            Priority: Minor
>         Attachments: S3-CSE Proposal.pdf
> This is for the proposal to introduce Client Side Encryption to S3 in such a way that
it can leverage HDFS transparent encryption, use the Hadoop KMS to manage keys, use the `hdfs
crypto` command line tools to manage encryption zones in the cloud, and enable distcp to copy
from HDFS to S3 (and vice-versa) with data still encrypted.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message