Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id BF75117977 for ; Tue, 24 Feb 2015 22:17:49 +0000 (UTC) Received: (qmail 95610 invoked by uid 500); 24 Feb 2015 22:17:43 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 95464 invoked by uid 500); 24 Feb 2015 22:17:43 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 95447 invoked by uid 99); 24 Feb 2015 22:17:43 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Feb 2015 22:17:43 +0000 X-ASF-Spam-Status: No, hits=1.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of kartha02@gmail.com designates 209.85.212.176 as permitted sender) Received: from [209.85.212.176] (HELO mail-wi0-f176.google.com) (209.85.212.176) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Feb 2015 22:17:18 +0000 Received: by mail-wi0-f176.google.com with SMTP id h11so29047061wiw.3 for ; Tue, 24 Feb 2015 14:16:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=Cg2HzV3DifY46GZ+fOlqInfhJiGnoiWK0TKcoNDhi6g=; b=YfTcdd81ITxu45t1qhxJpKeOKO9+FPfAA89oJG2VYpqDPdPSr+7vO3HBdiFc4tdQX+ EBBtEvQej6Ur5j6JF7+TSPXcejvI23Lqa5T8xMVGeG1Xy4JjqJdgSqlIrDayILOvEXkQ IX+4+r/g832IGenlBRTVMfb6QxwVvTpk7siNudPmO9HNElIzi0aH7QW/bSk7jj9ByW6F 19CPXJyNS1SVGxXuasCyGmRzcnCnVA0KlCmmqWTCMZ8ATbEgYs7NQqr8+gMRAgx60xkz TkuI8lHQP21VI2TgAZv0vRwt88OEvheYD/ahRB6r1lfQtQrlNKUbZh0dCEEERLQ2H1k4 1Dmw== MIME-Version: 1.0 X-Received: by 10.180.90.197 with SMTP id by5mr712039wib.70.1424816192424; Tue, 24 Feb 2015 14:16:32 -0800 (PST) Received: by 10.28.99.67 with HTTP; Tue, 24 Feb 2015 14:16:32 -0800 (PST) In-Reply-To: References: Date: Tue, 24 Feb 2015 14:16:32 -0800 Message-ID: Subject: Re: Encryption At Rest Question From: Rajesh Kartha To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=f46d0438eb6d050fd7050fdcdf10 X-Virus-Checked: Checked by ClamAV on apache.org --f46d0438eb6d050fd7050fdcdf10 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Thank you Olivier, I suppose with the first suggestion - locking the dir to be unreadable for other users, the HDFS permissions would kick in and prevent an unwarranted user to read them. However, I wanted to see the actual encrypted data so I used the second approach you suggested. With hadoop fsck /mysecureDir -files -blocks -locations get the blocks for the directory, then go to the data node and perform a cat to see cryptic data for those block. Regards, Rajesh On Tue, Feb 24, 2015 at 12:28 PM, Olivier Renault wrote: > You can try looking at it with a user who doesn=E2=80=99t have permissi= on to > the folder. An alternative is to check which block it is on Linux and > looking at the block using cat from a linux shell. > > Olivier > > > From: Rajesh Kartha > Reply-To: "user@hadoop.apache.org" > Date: Tuesday, 24 February 2015 19:47 > To: "user@hadoop.apache.org" > Cc: "hdfs-dev@hadoop.apache.org" > Subject: Re: Encryption At Rest Question > > I was trying out the Transparent data at rest encryption and was able > to setup the KMS, zones etc. and add > files to the zone. > > How do I confirm if the files I added to the encryption zone are > encrypted ? Is there a way to view > the raw file, a *hdfs fs -cat *shows me the actual contents of the files > since the datanode decrypts it > before sending it. > > Thanks, > Rajesh > > > On Fri, Feb 20, 2015 at 11:42 PM, Ranadip Chatterjee > wrote: > >> In case of SSL enabled cluster, the DEK will be encrypted on the wire >> by the SSL layer. >> >> In case of non-SSL enabled cluster, it is not. But the intercepter only >> gets the DEK and not the encrypted data, so the data is still safe. Only= if >> the intercepter also manages to gain access to the encrypted data block = and >> associate that with the corresponding DEK, then the data is compromised. >> Given that each HDFS file has a different DEK, the intercepter has to ga= in >> quite a bit of access before the data is compromised. >> >> On 18 February 2015 at 00:04, Plamen Jeliazkov < >> plamen.jeliazkov@wandisco.com> wrote: >> >>> Hey guys, >>> >>> I had a question about how the new file encryption work done primarily >>> in HDFS-6134. >>> >>> I was just curious, how is the DEK protected on the wire? >>> Particularly after the KMS decrypts the EDEK and returns it to the >>> client. >>> >>> Thanks, >>> -Plamen >>> >>> >>> >>> 5 reasons your Hadoop needs WANdisco >>> >>> >>> Listed on the London Stock Exchange: WAND >>> >>> >>> THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY >>> BE PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its >>> subsidiaries, ("WANdisco") does not waive any confidentiality or >>> privilege. If you are not the intended recipient, please notify us >>> immediately and destroy the message without disclosing its contents to >>> anyone. Any distribution, use or copying of this e-mail or the informa= tion >>> it contains by other than an intended recipient is unauthorized. The v= iews >>> and opinions expressed in this e-mail message are the author's own and = may >>> not reflect the views and opinions of WANdisco, unless the author is >>> authorized by WANdisco to express such views or opinions on its behalf. >>> All email sent to or from this address is subject to electronic storage= and >>> review by WANdisco. Although WANdisco operates anti-virus programs, it >>> does not accept responsibility for any damage whatsoever caused by viru= ses >>> being passed. >>> >> >> >> >> -- >> Regards, >> Ranadip Chatterjee >> > > --f46d0438eb6d050fd7050fdcdf10 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thank you Olivier,

I= suppose with the first suggestion - locking the dir to be unreadable for o= ther users, the HDFS permissions would
kick in and prevent an= unwarranted user to read them.
However, I wanted to see the actual enc= rypted data so I used the second approach you suggested. With hadoop fsck /= mysecureDir -files -blocks -locations get the blocks for the directory, the= n go to the data node and perform a cat to see cryptic data for those block= .

Regards,
Rajesh

On Tue, Feb 24, 2015 at 12:28 P= M, Olivier Renault <orenault@hortonworks.com> wrote:<= br>
You can try looking at it with a user who doesn=E2=80=99t have permiss= ion to the folder. An alternative is to check which block it is on Linux an= d looking at the block using cat from a linux shell.=C2=A0

Olivier


From: Rajesh Kartha <kartha02@gmail.com>
Reply-To: "user@hadoop.apache.org" &= lt;user@hadoop.= apache.org>
Date: Tuesday, 24 February 2015 19:= 47
To: "user@hadoop.apache.org" <user@hadoop.apache= .org>
Cc: "hdfs-dev@hadoop.apache.org"= <hdfs-d= ev@hadoop.apache.org>
Subject: Re: Encryption At Rest Que= stion

I was trying out the Transparent data at rest encryption and was able = to setup the KMS, zones etc. and add
files to the zone.

How do I confirm if the files I added to the encryption zone are encrypted = ? Is there a way to view
the raw file, a hdfs fs -cat shows me the actual contents of the fil= es since the datanode decrypts it
before sending it.

Thanks,
Rajesh


On Fri, Feb 20, 2015 at 11:42 PM, Ranadip Chatte= rjee <ranadip.c@gmai= l.com> wrote:
In case of SSL enabled cluster, the DEK will be encrypted on the wire = by the SSL layer.

In case of non-SSL enabled cluster, it is not. But the intercepter only get= s the DEK and not the encrypted data, so the data is still safe. Only if th= e intercepter also manages to gain access to the encrypted data block and a= ssociate that with the corresponding DEK, then the data is compromised. Given that each HDFS file has a differe= nt DEK, the intercepter has to gain quite a bit of access before the data i= s compromised.

On 18 February 2015 at 00:04, Plamen Jeliazkov <= span dir=3D"ltr"> <plam= en.jeliazkov@wandisco.com> wrote:
Hey guys,

I had a question about how the new file encryption work done primarily= in HDFS-6134.

I was just curious, how is the DEK protected on the wire?=C2=A0
Particularly after the KMS decrypts the EDEK and returns it to the cli= ent.

Thanks,
-Plamen



5 reasons your Hadoop needs WANdisco

Listed on the London Stock Exchange:=C2=A0WAND

THIS ME= SSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE PRIVILE= GED.=C2=A0 If this message was misdirected, WANdisco, Inc. and its subsidia= ries, ("WANdisco") does not waive any confidentiality or privilege.=C2=A0 If you are not the intended recipient, please notify u= s immediately and destroy the message without disclosing its contents to an= yone.=C2=A0 Any distribution, use or copying of this e-mail or the informat= ion it contains by other than an intended recipient is unauthorized.=C2=A0 The views and opinions expressed in this = e-mail message are the author's own and may not reflect the views and o= pinions of WANdisco, unless the author is authorized by WANdisco to express= such views or opinions on its behalf.=C2=A0 All email sent to or from this address is subject to electronic storage and re= view by WANdisco.=C2=A0 Although WANdisco operates anti-virus programs, it = does not accept responsibility for any damage whatsoever caused by viruses = being passed.




--
Regards,
Ranadip Chatterjee

--f46d0438eb6d050fd7050fdcdf10--