hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Encrypting files in Hadoop - Using the io.compression.codecs
Date Tue, 07 Aug 2012 08:02:11 GMT

The codec org.apache.hadoop.io.compress.crypto.CyptoCodec needs to be
used. What you've done so far is merely add it to be loaded by Hadoop
at runtime, but you will need to use it in your programs if you wish
for it to get applied.

For example, for MapReduce outputs to be compressed, you may run an MR
job with the following option set on its configuration:


And then you can notice that your output files were all properly
encrypted with the above codec.

Likewise, if you're using direct HDFS writes, you will need to wrap
your outputstream with this codec. Look at the CompressionCodec API to
see how: http://hadoop.apache.org/common/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodec.html#createOutputStream(java.io.OutputStream)
(Where your CompressionCodec must be the
org.apache.hadoop.io.compress.crypto.CyptoCodec instance).

On Tue, Aug 7, 2012 at 1:11 PM, Farrokh Shahriari
<mohandes.zebeleh.67@gmail.com> wrote:
> Hello
> I use "Hadoop Crypto Compressor" from this site"https://github.com/geisbruch/HadoopCryptoCompressor"
for encryption hdfs files.
> I've downloaded the complete code & create the jar file,Change the propertise in
core-site.xml as the site says.
> But when I add a new file,nothing has happened & encryption isn't working.
> What can I do for encryption hdfs files ? Does anyone know how I should use this class
> Tnx

Harsh J

View raw message