hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem
Date Thu, 14 Jul 2016 09:04:20 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15376617#comment-15376617
] 

Steve Loughran commented on HADOOP-13075:
-----------------------------------------

the get with range is a reference to how, after a seek, we do a GET with a range of pos-EOF
(default), or, with fadvise set to random, pos+block size.

I'm really curious what happens here: if we seek to an offset and then read, how long do those
reads take? I think it will depend on how the encryption is done: if it is done from byte
0 in some streaming cipher, then you would have to go through the whole file to decrypt later
bytes. If the blob is encrypted in separate blocks, then you'd have to go back a block...something
we could potentially work out.

Overall, I like the patch, it's pretty close to being ready to add

> Add support for SSE-KMS and SSE-C in s3a filesystem
> ---------------------------------------------------
>
>                 Key: HADOOP-13075
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13075
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Andrew Olson
>            Assignee: Federico Czerwinski
>
> S3 provides 3 types of server-side encryption [1],
> * SSE-S3 (Amazon S3-Managed Keys) [2]
> * SSE-KMS (AWS KMS-Managed Keys) [3]
> * SSE-C (Customer-Provided Keys) [4]
> Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 (HADOOP-10568)
-- the underlying aws-java-sdk makes that very simple [5]. With native support in aws-java-sdk
already available it should be fairly straightforward [6],[7] to support the other two types
of SSE with some additional fs.s3a configuration properties.
> [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html
> [2] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html
> [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html
> [4] http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html
> [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html
> [6] http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java
> [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message