hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dilli Arumugam (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5143) Hadoop cryptographic file system
Date Fri, 30 Aug 2013 21:24:53 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755152#comment-13755152

Dilli Arumugam commented on HDFS-5143:

Couple of questions:

Use case:
A client program has to read 2 files File1, File2.
File1 is ecnrypted.
File2 is not encrypted.

How is the client supposed to choose plain HDFS protocol versus CFS?
In other words, how would the client detect whether the file is encrypted?

Would this play nicely with hadoop command line 

hadoop fs -cat File1
hadoop fs -cat File2

I am wondering whether we should consider adding metadata to filesystem namespace, an attribute(s)
such as encrypted:Boolean, encryptionKeyAlias:String. With this approach, namenode could return
these attributes to authenticated and authorized client.  The client can look up the key from
keylookup service passing the keyAlias. Key look up service would do required authentication,
authroization checks on the client before returning the key. This of course requires changes
to core hadoop and have to be considered carefully against pros and cons.


> Hadoop cryptographic file system
> --------------------------------
>                 Key: HDFS-5143
>                 URL: https://issues.apache.org/jira/browse/HDFS-5143
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: security
>    Affects Versions: 3.0.0
>            Reporter: Yi Liu
>              Labels: rhino
>             Fix For: 3.0.0
>         Attachments: HADOOP cryptographic file system.pdf
> There is an increasing need for securing data when Hadoop customers use various upper
layer applications, such as Map-Reduce, Hive, Pig, HBase and so on.
> HADOOP CFS (HADOOP Cryptographic File System) is used to secure data, based on HADOOP
“FilterFileSystem” decorating DFS or other file systems, and transparent to upper layer
applications. It’s configurable, scalable and fast.
> High level requirements:
> 1.	Transparent to and no modification required for upper layer applications.
> 2.	“Seek”, “PositionedReadable” are supported for input stream of CFS if the
wrapped file system supports them.
> 3.	Very high performance for encryption and decryption, they will not become bottleneck.
> 4.	Can decorate HDFS and all other file systems in Hadoop, and will not modify existing
structure of file system, such as namenode and datanode structure if the wrapped file system
is HDFS.
> 5.	Admin can configure encryption policies, such as which directory will be encrypted.
> 6.	A robust key management framework.
> 7.	Support Pread and append operations if the wrapped file system supports them.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message