hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Brown <a...@hortonworks.com>
Subject Re: Encrypting files in Hadoop - Using the io.compression.codecs
Date Tue, 07 Aug 2012 17:53:30 GMT
Hi Farrokh,

what is your concern here? non hadoop processes accessing the data? or
hadoop users accessing data they are not supposed to?

-Adam

On Tue, Aug 7, 2012 at 1:13 AM, Farrokh Shahriari
<mohandes.zebeleh.67@gmail.com> wrote:
> Thanks,
> What if I want to use this encryption in a cluster with hbase running on top
> of hadoop? Can't hadoop be configured to automatically encrypt each file
> which is going to be written on it?
> If not I probably should be asking how to enable encryption on hbase, and
> asking this question on the hbase mailing list, right?
>
>
> On Tue, Aug 7, 2012 at 12:32 PM, Harsh J <harsh@cloudera.com> wrote:
>>
>> Farrokh,
>>
>> The codec org.apache.hadoop.io.compress.crypto.CyptoCodec needs to be
>> used. What you've done so far is merely add it to be loaded by Hadoop
>> at runtime, but you will need to use it in your programs if you wish
>> for it to get applied.
>>
>> For example, for MapReduce outputs to be compressed, you may run an MR
>> job with the following option set on its configuration:
>>
>>
>> "-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.crypto.CyptoCodec"
>>
>> And then you can notice that your output files were all properly
>> encrypted with the above codec.
>>
>> Likewise, if you're using direct HDFS writes, you will need to wrap
>> your outputstream with this codec. Look at the CompressionCodec API to
>> see how:
>> http://hadoop.apache.org/common/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodec.html#createOutputStream(java.io.OutputStream)
>> (Where your CompressionCodec must be the
>> org.apache.hadoop.io.compress.crypto.CyptoCodec instance).
>>
>> On Tue, Aug 7, 2012 at 1:11 PM, Farrokh Shahriari
>> <mohandes.zebeleh.67@gmail.com> wrote:
>> >
>> > Hello
>> > I use "Hadoop Crypto Compressor" from this
>> > site"https://github.com/geisbruch/HadoopCryptoCompressor" for encryption
>> > hdfs files.
>> > I've downloaded the complete code & create the jar file,Change the
>> > propertise in core-site.xml as the site says.
>> > But when I add a new file,nothing has happened & encryption isn't
>> > working.
>> > What can I do for encryption hdfs files ? Does anyone know how I should
>> > use this class ?
>> >
>> > Tnx
>>
>>
>>
>>
>> --
>> Harsh J
>
>



-- 
Adam Brown
Enablement Engineer
Hortonworks

Mime
View raw message