From user-return-252-apmail-hadoop-user-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 10 06:46:38 2012 Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id BD17EDD1C for ; Fri, 10 Aug 2012 06:46:38 +0000 (UTC) Received: (qmail 43252 invoked by uid 500); 10 Aug 2012 06:46:34 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 42800 invoked by uid 500); 10 Aug 2012 06:46:32 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 42773 invoked by uid 99); 10 Aug 2012 06:46:31 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Aug 2012 06:46:31 +0000 X-ASF-Spam-Status: No, hits=1.8 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of mohandes.zebeleh.67@gmail.com designates 209.85.216.41 as permitted sender) Received: from [209.85.216.41] (HELO mail-qa0-f41.google.com) (209.85.216.41) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Aug 2012 06:46:25 +0000 Received: by qafk30 with SMTP id k30so868430qaf.14 for ; Thu, 09 Aug 2012 23:46:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=6ZMbuN5ttBLvXkcH9N9yua0Obty9JzoZNaYixDmM5cY=; b=0XosIzYJ2dMoCGgVnAuzB0PnOxqNPOUN+onuAKKOhl6lb+f3akvB9n4a5yTn/dh4eC ogD6MV+1ZASrmL23qWsRq1tWTQy16KO6Pg+nukAFzENdwbq2rjyD15m4rUl951QZGHvX kOuE+/OyaeI6eS30cMpF//iKfanCPzDEhZVHZiclkPuz+SzHyDmDUOY9BrM+wUYcuz5p oHkUlOwxMoWHO/ESuVsokURbLpfmPt5gvFkyCJATXgD4oxGimQCMsI0+VhCzuYRbwKSr 0ykK5uRxp4LS9+kk39ELykvsiINJf46uFwht7wwN5mR2fvTEuzIqYycBEq+bpc66q3OZ 9R8Q== MIME-Version: 1.0 Received: by 10.229.135.147 with SMTP id n19mr1057972qct.11.1344581164700; Thu, 09 Aug 2012 23:46:04 -0700 (PDT) Received: by 10.49.132.167 with HTTP; Thu, 9 Aug 2012 23:46:04 -0700 (PDT) In-Reply-To: References: Date: Fri, 10 Aug 2012 11:16:04 +0430 Message-ID: Subject: Re: Encrypting files in Hadoop - Using the io.compression.codecs From: Farrokh Shahriari To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=00248c769122b1939804c6e3b225 --00248c769122b1939804c6e3b225 Content-Type: text/plain; charset=ISO-8859-1 Hi Adam, Beause I have very important data,I wanna use encryption for preventing physical access of other users. Michael,how can I use coprocessor for encryption? Is there any function for doing that ? Thanks for helping me On Tue, Aug 7, 2012 at 10:23 PM, Adam Brown wrote: > Hi Farrokh, > > what is your concern here? non hadoop processes accessing the data? or > hadoop users accessing data they are not supposed to? > > -Adam > > On Tue, Aug 7, 2012 at 1:13 AM, Farrokh Shahriari > wrote: > > Thanks, > > What if I want to use this encryption in a cluster with hbase running on > top > > of hadoop? Can't hadoop be configured to automatically encrypt each file > > which is going to be written on it? > > If not I probably should be asking how to enable encryption on hbase, and > > asking this question on the hbase mailing list, right? > > > > > > On Tue, Aug 7, 2012 at 12:32 PM, Harsh J wrote: > >> > >> Farrokh, > >> > >> The codec org.apache.hadoop.io.compress.crypto.CyptoCodec needs to be > >> used. What you've done so far is merely add it to be loaded by Hadoop > >> at runtime, but you will need to use it in your programs if you wish > >> for it to get applied. > >> > >> For example, for MapReduce outputs to be compressed, you may run an MR > >> job with the following option set on its configuration: > >> > >> > >> > "-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.crypto.CyptoCodec" > >> > >> And then you can notice that your output files were all properly > >> encrypted with the above codec. > >> > >> Likewise, if you're using direct HDFS writes, you will need to wrap > >> your outputstream with this codec. Look at the CompressionCodec API to > >> see how: > >> > http://hadoop.apache.org/common/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodec.html#createOutputStream(java.io.OutputStream) > >> (Where your CompressionCodec must be the > >> org.apache.hadoop.io.compress.crypto.CyptoCodec instance). > >> > >> On Tue, Aug 7, 2012 at 1:11 PM, Farrokh Shahriari > >> wrote: > >> > > >> > Hello > >> > I use "Hadoop Crypto Compressor" from this > >> > site"https://github.com/geisbruch/HadoopCryptoCompressor" for > encryption > >> > hdfs files. > >> > I've downloaded the complete code & create the jar file,Change the > >> > propertise in core-site.xml as the site says. > >> > But when I add a new file,nothing has happened & encryption isn't > >> > working. > >> > What can I do for encryption hdfs files ? Does anyone know how I > should > >> > use this class ? > >> > > >> > Tnx > >> > >> > >> > >> > >> -- > >> Harsh J > > > > > > > > -- > Adam Brown > Enablement Engineer > Hortonworks > --00248c769122b1939804c6e3b225 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi Adam,
Beause I have very important data,I wanna use encryption for pr= eventing physical access of other users.
Michael,how can I use coprocess= or for encryption? Is there any function for doing that ?

Thanks for= helping me

On Tue, Aug 7, 2012 at 10:23 PM, Adam Brown = <adam@hortonworks.com> wrote:
Hi Farrokh,

what is your concern here? non hadoop processes accessing the data? or
hadoop users accessing data they are not supposed to?

-Adam

On Tue, Aug 7, 2012 at 1:13 AM, Farrokh Shahriari
<mohandes.zebeleh.67@gm= ail.com> wrote:
> Thanks,
> What if I want to use this encryption in a cluster with hbase running = on top
> of hadoop? Can't hadoop be configured to automatically encrypt eac= h file
> which is going to be written on it?
> If not I probably should be asking how to enable encryption on hbase, = and
> asking this question on the hbase mailing list, right?
>
>
> On Tue, Aug 7, 2012 at 12:32 PM, Harsh J <harsh@cloudera.com> wrote:
>>
>> Farrokh,
>>
>> The codec org.apache.hadoop.io.compress.crypto.CyptoCodec needs to= be
>> used. What you've done so far is merely add it to be loaded by= Hadoop
>> at runtime, but you will need to use it in your programs if you wi= sh
>> for it to get applied.
>>
>> For example, for MapReduce outputs to be compressed, you may run a= n MR
>> job with the following option set on its configuration:
>>
>>
>> "-Dmapred.output.compression.codec=3Dorg.apache.hadoop.io.com= press.crypto.CyptoCodec"
>>
>> And then you can notice that your output files were all properly >> encrypted with the above codec.
>>
>> Likewise, if you're using direct HDFS writes, you will need to= wrap
>> your outputstream with this codec. Look at the CompressionCodec AP= I to
>> see how:
>> http://hadoop.apache.org/common/docs/stab= le/api/org/apache/hadoop/io/compress/CompressionCodec.html#createOutputStre= am(java.io.OutputStream)
>> (Where your CompressionCodec must be the
>> org.apache.hadoop.io.compress.crypto.CyptoCodec instance).
>>
>> On Tue, Aug 7, 2012 at 1:11 PM, Farrokh Shahriari
>> <mohandes.zebe= leh.67@gmail.com> wrote:
>> >
>> > Hello
>> > I use "Hadoop Crypto Compressor" from this
>> > site"https://github.com/geisbruch/HadoopCryptoCom= pressor" for encryption
>> > hdfs files.
>> > I've downloaded the complete code & create the jar fi= le,Change the
>> > propertise in core-site.xml as the site says.
>> > But when I add a new file,nothing has happened & encrypti= on isn't
>> > working.
>> > What can I do for encryption hdfs files ? Does anyone know ho= w I should
>> > use this class ?
>> >
>> > Tnx
>>
>>
>>
>>
>> --
>> Harsh J
>
>



--
Adam Brown
Enablement Engineer
Hortonworks

--00248c769122b1939804c6e3b225--