hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15006) Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS
Date Thu, 11 Jan 2018 19:54:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16322852#comment-16322852
] 

Steve Loughran commented on HADOOP-15006:
-----------------------------------------

Look how SequenceFile.Reader() works: it gets the length of the file from getFileStatus()
& then uses it downstream. If its size != stream length, this is the code which crashes
first :)
  

Imagine we had 
  
{code}  

FSDataInputStream file = openFile(fs, filename, bufSize, len);
len = file.getLength();
...
{code}

Fix that class and Hadoop internally gets robust, and on object stores, actually cuts out
a HEAD request. (saves $0.005 and 100 mS).
Patch ORC & Parquet and you've just moved the core formats onto it too.


> Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS
> ---------------------------------------------------------------
>
>                 Key: HADOOP-15006
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15006
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs/s3, kms
>            Reporter: Steve Moist
>            Priority: Minor
>         Attachments: S3-CSE Proposal.pdf
>
>
> This is for the proposal to introduce Client Side Encryption to S3 in such a way that
it can leverage HDFS transparent encryption, use the Hadoop KMS to manage keys, use the `hdfs
crypto` command line tools to manage encryption zones in the cloud, and enable distcp to copy
from HDFS to S3 (and vice-versa) with data still encrypted.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message