Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of java8964@hotmail.com
 designates 65.55.111.82 as permitted sender)
Message-ID: <BLU162-W1031D3EEB78CCCF4B27FD6D0FC0@phx.gbl>
Content-Type: multipart/alternative;
	boundary="_26c6df2f-d8fa-4652-8aaf-32d3b0638bc5_"
From: java8964 java8964 <java8964@hotmail.com>
To: <user@hadoop.apache.org>
Subject: RE: Encryption in HDFS
Date: Tue, 26 Feb 2013 14:52:09 -0500
Importance: Normal
In-Reply-To: 
 <CALbKeSjcKAvykwDB5qS_EQr39tnnmtUQiZ4ENpiCjwGWPvOPzQ@mail.gmail.com>
References: 
 <CALbKeSjcKAvykwDB5qS_EQr39tnnmtUQiZ4ENpiCjwGWPvOPzQ@mail.gmail.com>
MIME-Version: 1.0

--_26c6df2f-d8fa-4652-8aaf-32d3b0638bc5_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


I am also interested in your research. Can you share some insight about the=
 following questions?
1) When you use CompressionCodec=2C can the encrypted file split? From my u=
nderstand=2C there is no encrypt way can make the file decryption individua=
lly by block=2C right?  For example=2C if I have 1G file=2C encrypted using=
 AES=2C how do you or can you decrypt the file block by block=2C instead of=
 just using one mapper to decrypt the whole file?
2) In your CompressionCodec implementation=2C do you use the DecompressorSt=
ream or BlockDecompressorStream? If BlockDecompressorStream=2C can you shar=
e some examples? Right now=2C I have some problems to use BlockDecompressor=
Stream to do the exactly same thing as you did.3) Do you have any plan to s=
hare your code=2C especially if you did use BlockDecompressorStream and mak=
e the encryption file decrypted block by block in the hadoop MapReduce job.
Thanks
Yong
From: renderaid@gmail.com
Date: Tue=2C 26 Feb 2013 14:10:08 +0900
Subject: Encryption in HDFS
To: user@hadoop.apache.org

Hello=2C I'm a university student.
I implemented AES and Triple DES with CompressionCodec in java cryptography=
 architecture (JCA)The encryption is performed by a client node using Hadoo=
p API.

Map tasks read blocks from HDFS and these blocks are decrypted by each map =
tasks.I tested my implementation with generic HDFS. My cluster consists of =
3 nodes (1 master node=2C 3 worker nodes) and each machines have quad core =
processor (i7-2600) and 4GB memory.=20

A test input is 1TB text file which consists of 32 multiple text files (1 t=
ext file is 32GB)
I expected that the encryption takes much more time than generic HDFS. The =
performance does not differ significantly.=20

The decryption step takes about 5-7% more than generic HDFS. The encryption=
 step takes about 20-30% more than generic HDFS because it is implemented b=
y single thread and executed by 1 client node.=20

So the encryption can get more performance.=20
May there be any error in my test?
I know there are several implementation for encryting files in HDFS. Are th=
ese implementations enough to secure HDFS?


best regards=2C
seonpark
* Sorry for my bad english  		 	   		  =

--_26c6df2f-d8fa-4652-8aaf-32d3b0638bc5_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<html>
<head>
<style><!--
.hmmessage P
{
margin:0px=3B
padding:0px
}
body.hmmessage
{
font-size: 10pt=3B
font-family:Tahoma
}
--></style></head>
<body class=3D'hmmessage'><div dir=3D'ltr'>
I am also interested in your research. Can you share some insight about the=
 following questions?<div><br></div><div>1) When you use&nbsp=3BCompression=
Codec=2C can the encrypted file split? From my understand=2C there is no en=
crypt way can make the file&nbsp=3Bdecryption&nbsp=3Bindividually&nbsp=3Bby=
 block=2C right?&nbsp=3B&nbsp=3BFor example=2C if I have 1G file=2C encrypt=
ed using AES=2C how do you or can you&nbsp=3Bdecrypt&nbsp=3Bthe file block =
by block=2C instead of just using one mapper to decrypt the whole file?<br>=
2) In your&nbsp=3BCompressionCodec implementation=2C do you use the&nbsp=3B=
DecompressorStream or&nbsp=3B<span style=3D"font-size: 10pt=3B">BlockDecomp=
ressorStream? If&nbsp=3B</span>BlockDecompressorStream=2C can you share som=
e examples? Right now=2C I have some problems to use&nbsp=3BBlockDecompress=
orStream to do the exactly same thing as you did.</div><div>3) Do you have =
any plan to share your code=2C especially if you did use BlockDecompressorS=
tream and make the encryption file decrypted block by block in the hadoop M=
apReduce job.</div><div><br></div><div>Thanks</div><div><br></div><div>Yong=
</div><div><br><div><div id=3D"SkyDrivePlaceholder"></div><hr id=3D"stopSpe=
lling">From: renderaid@gmail.com<br>Date: Tue=2C 26 Feb 2013 14:10:08 +0900=
<br>Subject: Encryption in HDFS<br>To: user@hadoop.apache.org<br><br><div>H=
ello=2C I'm a university student.</div><div><br></div><div>I implemented AE=
S and Triple DES with CompressionCodec in java cryptography architecture (J=
CA)</div><div>The encryption is performed by a client node using Hadoop API=
.</div>

<div>Map tasks read blocks from HDFS and these blocks are decrypted by each=
 map tasks.</div><div>I tested my implementation with generic HDFS.&nbsp=3B=
</div><div>My cluster consists of 3 nodes (1 master node=2C 3 worker nodes)=
 and each machines have quad core processor (i7-2600) and 4GB memory.&nbsp=
=3B</div>

<div>A test input is 1TB text file which consists of 32 multiple text files=
 (1 text file is 32GB)</div><div><br></div><div>I expected that the encrypt=
ion takes much more time than generic HDFS.&nbsp=3B</div><div>The performan=
ce does not differ significantly.&nbsp=3B</div>

<div>The decryption step takes about 5-7% more than generic HDFS.&nbsp=3B</=
div><div>The encryption step takes about 20-30% more than generic HDFS beca=
use it is implemented by single thread and executed by 1 client node.&nbsp=
=3B</div><div>

So the encryption can get more performance.&nbsp=3B</div><div><br></div><di=
v>May there be any error in my test?</div><div><br></div><div>I know there =
are several implementation for encryting files in HDFS.&nbsp=3B</div><div>A=
re these implementations enough to secure HDFS?</div>

<div><br></div><div>best regards=2C</div><div><br></div><div>seonpark</div>=
<div><br></div><div>* Sorry for my bad english&nbsp=3B</div><BR><BR></div><=
/div> 		 	   		  </div></body>
</html>=

--_26c6df2f-d8fa-4652-8aaf-32d3b0638bc5_--