Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6180DED97 for ; Tue, 26 Feb 2013 05:25:17 +0000 (UTC) Received: (qmail 28091 invoked by uid 500); 26 Feb 2013 05:25:12 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 27779 invoked by uid 500); 26 Feb 2013 05:25:12 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 27750 invoked by uid 99); 26 Feb 2013 05:25:11 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Feb 2013 05:25:11 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of yuzhihong@gmail.com designates 209.85.161.179 as permitted sender) Received: from [209.85.161.179] (HELO mail-gg0-f179.google.com) (209.85.161.179) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Feb 2013 05:25:04 +0000 Received: by mail-gg0-f179.google.com with SMTP id h4so626397ggn.24 for ; Mon, 25 Feb 2013 21:24:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=90ASiOwcRM2RHiPtTD2wWokUjeIvyE/xHDoJFkPc3os=; b=XrqbfTpGsZU11GAQQUGvOiRlv3DzUZpqrLGCVhhCrwZPj6Av6Bry3LWrjxgp01gBH6 AlQ3BdGb+BeGbmmwzZkSgwvKmarSmStTmJKXairpKRGjshJKQyI2fcEhgCBC9gqdRJW4 nm5ozjE5YcTB0A5XDkkcwj484uZzjQD5AlQFBH3ahMkN/ED4UCKQG79Pdkk4+1l6kQDB wG4G+jE+T/SbuClEFKTcEBYI2pVQuhvIU4ByQ1dGvsO6Z7vTjwjkGCeudjNlr0bqcDhb uxjD9qZInZCXcLEPMU89PAQy68j5DvVs61WzFyCVPRYyw5IFHB8E11W1PEAxP77tgYqV gvig== MIME-Version: 1.0 X-Received: by 10.236.79.9 with SMTP id h9mr22636257yhe.83.1361856283808; Mon, 25 Feb 2013 21:24:43 -0800 (PST) Received: by 10.101.208.18 with HTTP; Mon, 25 Feb 2013 21:24:43 -0800 (PST) In-Reply-To: References: Date: Mon, 25 Feb 2013 21:24:43 -0800 Message-ID: Subject: Re: Encryption in HDFS From: Ted Yu To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=20cf30050dfc083a3204d699e077 X-Virus-Checked: Checked by ClamAV on apache.org --20cf30050dfc083a3204d699e077 Content-Type: text/plain; charset=ISO-8859-1 The following JIRAs are related to your research: HADOOP-9331: Hadoop crypto codec framework and crypto codec implementations< https://issues.apache.org/jira/browse/hadoop-9331> and related sub-tasks MAPREDUCE-5025: Key Distribution and Management for supporting crypto codec in Map Reduce and related JIRAs On Mon, Feb 25, 2013 at 9:10 PM, Seonyeong Bak wrote: > Hello, I'm a university student. > > I implemented AES and Triple DES with CompressionCodec in java > cryptography architecture (JCA) > The encryption is performed by a client node using Hadoop API. > Map tasks read blocks from HDFS and these blocks are decrypted by each map > tasks. > I tested my implementation with generic HDFS. > My cluster consists of 3 nodes (1 master node, 3 worker nodes) and each > machines have quad core processor (i7-2600) and 4GB memory. > A test input is 1TB text file which consists of 32 multiple text files (1 > text file is 32GB) > > I expected that the encryption takes much more time than generic HDFS. > The performance does not differ significantly. > The decryption step takes about 5-7% more than generic HDFS. > The encryption step takes about 20-30% more than generic HDFS because it > is implemented by single thread and executed by 1 client node. > So the encryption can get more performance. > > May there be any error in my test? > > I know there are several implementation for encryting files in HDFS. > Are these implementations enough to secure HDFS? > > best regards, > > seonpark > > * Sorry for my bad english > > --20cf30050dfc083a3204d699e077 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
The following JIRAs are related to your research:

= HADOOP-9331: Hadoop crypto codec fra= mework and crypto codec implementations<https://issues.apache.org/jira/browse/hadoop-9331> and related sub-tasks

MAPREDUCE-5025: Key Distribution and Management for supporting crypto code= c in Map Reduce<https:/= /issues.apache.org/jira/browse/mapreduce-5025> and related JIRAs

On Mon, Feb 25, 2013 at 9:10 PM, Seonyeong B= ak <renderaid@gmail.com> wrote:
Hello, I'm a university student.

I implem= ented AES and Triple DES with CompressionCodec in java cryptography archite= cture (JCA)
The encryption is performed by a client node using Ha= doop API.
Map tasks read blocks from HDFS and these blocks are decrypted by each= map tasks.
I tested my implementation with generic HDFS.=A0
My cluster consists of 3 nodes (1 master node, 3 worker nodes) and ea= ch machines have quad core processor (i7-2600) and 4GB memory.=A0
A test input is 1TB text file which consists of 32 multiple text files= (1 text file is 32GB)

I expected that the encrypt= ion takes much more time than generic HDFS.=A0
The performance do= es not differ significantly.=A0
The decryption step takes about 5-7% more than generic HDFS.=A0
<= div>The encryption step takes about 20-30% more than generic HDFS because i= t is implemented by single thread and executed by 1 client node.=A0
So the encryption can get more performance.=A0

May= there be any error in my test?

I know there are s= everal implementation for encryting files in HDFS.=A0
Are these i= mplementations enough to secure HDFS?

best regards,

seonpark

* Sorry for my bad english=A0


--20cf30050dfc083a3204d699e077--