hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jerry Chen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5025) Key Distribution and Management for supporting crypto codec in Map Reduce
Date Tue, 26 Feb 2013 02:58:14 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13586660#comment-13586660

Jerry Chen commented on MAPREDUCE-5025:

The reason we proposed file formats extended with encryption support instead of something
at a lower layer is so the user can selectively apply encryption only where they feel necessary,
only for those MR jobs that require it. This also helps keep the resulting files and the usage
of encryption filesystem agnostic.

Adding transparent encryption to a filesystem is an interesting idea and something that we
also prototyped as part of this work. Perhaps a Common JIRA for an encrypting filesystem derived
from FileSystem would be appropriate? Or an HDFS JIRA  for plugging in compression and crypto
codecs to block storage and transfer? We could look at something like those for follow on

> Key Distribution and Management for supporting crypto codec in Map Reduce
> -------------------------------------------------------------------------
>                 Key: MAPREDUCE-5025
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5025
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: security
>    Affects Versions: trunk
>            Reporter: Jerry Chen
>            Assignee: Jerry Chen
>         Attachments: MAPREDUCE-5025.patch
>   Original Estimate: 504h
>  Remaining Estimate: 504h
> This task defines the work to enable Map Reduce to utilize the Crypto Codec framework
to support encryption and decryption of data during MapReduce Job.
> According to the some real use case and discussions from the community, for encryption
and decryption files in Map Reduce, we have the following requirements:
>   1. Different stages (input, output, intermediate output) should have the flexibility
to choose whether encrypt or not, as well as which crypto codec to use.
>   2. Different stages may have different scheme of providing the keys.
>   3. Different Files (for example, different input files) may have or use different keys.

>   4. Support a flexible way of retrieving keys for encryption or decryption.
> So this task defines and provides the framework for supporting these requirements as
well as the implementations for common use and key retrieving scenarios.
> The design document of this part is included in the Hadoop Crypto Design attached in

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message