Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-dev@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of sanjay@hortonworks.com
 designates 209.85.192.170 as permitted sender)
Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\))
Subject: Re: [VOTE] Merge fs-encryption branch to trunk
From: sanjay Radia <sanjay@hortonworks.com>
In-Reply-To: <7418FD52-E794-4DEE-B6B1-E0D5E162E8FD@hortonworks.com>
Date: Fri, 15 Aug 2014 15:13:29 -0700
Cc: "common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>
Message-Id: <3515A87E-A328-41E9-8368-F68BE31B00FC@hortonworks.com>
References: 
 <CAGB5D2YoX1hPtBL56D-LD3wbM3VnZgkfOm0dFNwB9bXVxqJkRg@mail.gmail.com>
 <7418FD52-E794-4DEE-B6B1-E0D5E162E8FD@hortonworks.com>
To: hdfs-dev <hdfs-dev@hadoop.apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable


+1 (binding)
We have made some great progress in the last few days on some of the issues=
 I raised.
I have posted a summary of the followup items that are needed on the Jira t=
oday.
I am +1ing expecting the team will  complete Items 1 (distcp/cp) and 2 (web=
hdfs)  promptly. Before we publish transparent encryption in a 2.x release =
for pubic consumption, let us at least complete item 1 (ie distcp and cp) a=
nd the flag to turn this feature on/of.

This is a great work; thanks team for contributing this important feature.

sanjay

On Aug 14, 2014, at 1:05 AM, sanjay Radia <sanjay@hortonworks.com> wrote:

> While I was originally skeptical of transparent encryption, I like the va=
lue proposition of transparent encryption. HDFS has several layers, protoco=
ls  and tools. While the HDFS core part seems to be well done in the Jira, =
inserting the matching transparency in the other tools or protocols need to=
 be worked through.
>=20
> I have the following areas of concern:
> - Common protocols like webhdfs should continue to work (the design doc m=
arks this as a goal), This issue is being discussed in the Jira but it appe=
ars that webhdfs does not currently work with encrypted files: Andrew say t=
hat "Regarding webhdfs, it's not a recommended deployment" and that he will=
 modify the documentation to match that. Aljeandro say "Both httpfs and web=
hdfs will work just fine" but then in the same paragraph says "this could f=
ail some security audits". We need to resolve this quickly. Webhdfs is heav=
ily used by many Hadoop users.
>=20
>=20
> - Common tools should like cp, distcp and HAR should continue  to work wi=
th non-encrypted and encrypted files in an automatic fashion. This issue ha=
s been heavily discussed in the Jira and at the meeting. The /.reserved./.r=
aw mechanism appears to be a step in the right direction for distcp and cp,=
 however this work has not reached its conclusion in my opinion; Charles ar=
e I are going through the use cases and I think we are close to a clean sol=
ution for distcp and cp.  HAR still needs a concrete proposal.
>=20
> - KMS scalability in medium to large clusters. This can perhaps  be addre=
ssed by getting the keys ahead of time when a job is submitted.  Without th=
is the  KMS will need to be as highly available and scalable as the NN.  I =
think this is future implementation work but we need to at least determine =
if this is indeed possible in case we need to modify some of the APIs right=
 now to support that.
>=20
> There are some other minor things under discussion, and I still need to g=
o through the new APIs.
>=20
> Unfortunately at this stage I cannot give a +1 for this merge; I hope to =
change this in the next day or -  I am working with the Jira's team.  Aleja=
ndoro, Charles, Andrew, Atm, ...  to resolve the above as quickly as possib=
le.
>=20
> Sanjay (binding)
>=20
>=20
>=20
> On Aug 8, 2014, at 11:45 AM, Andrew Wang <andrew.wang@cloudera.com> wrote=
:
>=20
>> Hi all,
>>=20
>> I'd like to call a vote to merge the fs-encryption branch to trunk.
>> Development of this feature has been ongoing since March on HDFS-6134 an=
d
>> HADOOP-10150, totally approximately 50 commits.
>>=20
>> .....
>> Thanks,
>> Andrew
>=20


--=20
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to=
=20
which it is addressed and may contain information that is confidential,=20
privileged and exempt from disclosure under applicable law. If the reader=
=20
of this message is not the intended recipient, you are hereby notified that=
=20
any printing, copying, dissemination, distribution, disclosure or=20
forwarding of this communication is strictly prohibited. If you have=20
received this communication in error, please contact the sender immediately=
=20
and delete it from your system. Thank You.