hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem
Date Fri, 27 Jan 2017 10:06:25 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15842482#comment-15842482

Steve Loughran commented on HADOOP-13075:

Don't worry about breaking our tests, those we can adapt if we are confident the new behaviour
is OK. That test was designed to implicitly verify that the encryption worked by setting an
invalid one. Otherwise, we had no evidence whatsoever that the property was being passed down.

You can just tweak that test to catch the error in initialization. That way we can verify
that invalid options are being rejected.

This does raise the question of how best to test the encryption work

# we should be able to verify that data written with one key cannot be parsed if a different
fs + key is used to read it.
# we should see what happens if you try to read unencrypted data with an FS with encryption
# maybe:  if a bucket is set up to require encryption, then unencrypted data cannot be written,
encrypted can. This implies that the tester will need a special bucket for this test &
declare it in the configs.

Something to try (now?) is for you to enable encryption in auth-keys.xml and rerun the full
test suite with encryption enabled. We've had hints in the past you may end up with a different
length of data returned in getFileStatus() than you get when you read in the data, and suspect
that it's related to encryption —though probably client-side, rather than server side. running
the entire aws test suite with encryption turned on helps check these things.

> Add support for SSE-KMS and SSE-C in s3a filesystem
> ---------------------------------------------------
>                 Key: HADOOP-13075
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13075
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Andrew Olson
>            Assignee: Federico Czerwinski
> S3 provides 3 types of server-side encryption [1],
> * SSE-S3 (Amazon S3-Managed Keys) [2]
> * SSE-KMS (AWS KMS-Managed Keys) [3]
> * SSE-C (Customer-Provided Keys) [4]
> Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 (HADOOP-10568)
-- the underlying aws-java-sdk makes that very simple [5]. With native support in aws-java-sdk
already available it should be fairly straightforward [6],[7] to support the other two types
of SSE with some additional fs.s3a configuration properties.
> [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html
> [2] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html
> [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html
> [4] http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html
> [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html
> [6] http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java
> [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message