Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-dev@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of vrsitaramanietflists@gmail.com
 designates 209.85.223.180 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAGCyb54QaTiC=kT6KTaboXmr0qxayayND50Ez5SpD5wqeqaNag@mail.gmail.com>
References: 
 <CAMnjchmndNOn3o+ane-MsMxoX_Uv3ZqEu3b5_ESxccHrOgtmSw@mail.gmail.com>
	<CAHtSEE8-HN8ePa4qnEQDgpUHp3uCDD9hQO=HSKp+UeXg+pCeeg@mail.gmail.com>
	<CAMnjchnOr=zJep3oVgHe5v3J87VEh9Ne1FB-Sqjpd-jM6N8wYw@mail.gmail.com>
	<CAGCyb54QaTiC=kT6KTaboXmr0qxayayND50Ez5SpD5wqeqaNag@mail.gmail.com>
Date: Sun, 14 Jun 2015 14:07:29 +0530
Message-ID: 
 <CAMnjchnYD97nvG33nHaPZSkfaTE+Lxf8ozV0Vz+Hp03aw-QPoQ@mail.gmail.com>
Subject: Re: Key Rotation in Data-at-Rest Encryption
From: Sitaraman Vilayannur <vrsitaramanietflists@gmail.com>
To: hdfs-dev@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001a113fefde6482e0051876409d

--001a113fefde6482e0051876409d
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi Arun,
 Thanks for your patience. I have a related question In my application i
need to encrypt/decrypt files
from the map reduce phase and i need to support key rotation.  Can i access
the KMS from the map/reduce
phase to retrieve the key material from the key alias which i retrieve from
the FileEncryptionInfo class stored in the
 extended attribute of the file(for decryption)?
 Any pointers to how i can plug in various key providers such as java
keystore would be appreciated.
Is the toString method used to store the FileEncryptionInfo into the
extended attribute if so how is the
FileEncryptionInfo object retrieved back from the String.
Thanks again for your help.
Sitaraman

On Sun, Jun 14, 2015 at 12:53 PM, Arun Suresh <asuresh@apache.org> wrote:

> Apologize if I wasn't clear
>
> > Is the EZ key version same as an alias for the key?
> yup
>
> > the EDEK along with the EZ key version is stored in the FIleInfo
> FileInfo contains both EDEK and EZ key version. The FileInfo (you can loo=
k
> at the *org.apache.hadoop.fs.FileEncryptionInfo* class for more info)
> object is stored as the value of the extended attribute of that file.
>
> > How is the KeyMaterial derived from the KeyAlias and where is the
> mapping between
> the two stored? Is it in the KMS?
> Yup. KMS extends the *org.apache.hadoop.crypto.key.KeyProvider* class. Yo=
u
> can take a look at it or a concrete implementation such as
> JavaKeyStoreProvider for more information.
>
> Also, you should probably direct questions related to HDFS encryption to
> hdfs-dev@hadoop.apache.org
>
> Cheers
> -Arun
>
>
>
> On Sun, Jun 14, 2015 at 12:11 AM, Sitaraman Vilayannur <
> vrsitaramanietflists@gmail.com> wrote:
>
> > Hi Arun,
> > Thanks for your response.
> > Could you explain this a bit further for me....
> > Is the EZ key version same as an alias for the key?
> > The EDEK is stored in the extended attributes of the file and the EZkey
> > Version is stored
> >  in the FileInfo  why is the EZKey Version not stored in the extended
> > attributes too.
> > Where is the FileInfo object persisted? Is it in the NameNode?
> > How is the KeyMaterial derived from the KeyAlias and where is the mappi=
ng
> > between the two stored? Is it in the KMS?
> > Thanks much for your help in this.
> > Sitaraman
> >
> > On Sun, Jun 14, 2015 at 12:14 PM, Arun Suresh <asuresh@cloudera.com>
> > wrote:
> >
> > > Hello Sitaraman,
> > >
> > > It is the EZ key "version" that is used to generate the EDEK (and whi=
ch
> > is
> > > ultimately stored in the encrypted file's extended attributes
> > > '*raw.hdfs.crypto.encryption.info
> > > <http://raw.hdfs.crypto.encryption.info>*'), not really the the EZ ke=
y
> > > itself (which is stored in the directory's extended attribute =E2=80=
=98
> > > *raw.hdfs.crypto.encryption.zone*=E2=80=99).
> > >
> > > Essentially, each file in a directory has a unique EDEK.. and an EDEK
> is
> > is
> > > generated with the current version of the directory EZ key. The EDEK
> > along
> > > with the EZ key version is stored in the FIleInfo. While decrypting,
> both
> > > these are passed on to the KMS which provides the client with the DEK
> > which
> > > can be used to decrypt the file.
> > >
> > > Hope this clarifies things.
> > >
> > > Cheers
> > > -Arun
> > >
> > > On Sat, Jun 13, 2015 at 9:51 PM, Sitaraman Vilayannur <
> > > vrsitaramanietflists@gmail.com> wrote:
> > >
> > > > HDFSDataatRestEncryption.pdf says the following about key
> > > rotation..(please
> > > > see appended below at the end of the mail)
> > > > If the existing files do not have their EDEKs reencrypted using the
> new
> > > > ezkeyid, how would the existing files be decrypted? That is where i=
s
> > the
> > > > mapping between files and its EZKey (for after key rotation differe=
nt
> > > files
> > > > have different EZKeys)ids stored and how is it retrieved?
> > > > Thanks
> > > > Sitaraman
> > > >
> > > > Key Rotation
> > > > When the administrator causes a key rotation of the EZkey
> > > > in the KMS, the encryption zone=E2=80=99s EZkey
> > > > (stored in the encryption zone directory=E2=80=99s
> > > raw.hdfs.crypto.encryption.zone
> > > > extended attribute) gets the new keyid and version (only the versio=
n
> > > > changes). Any new files
> > > > created in the encryption zone have their DEKs encrypted using the
> new
> > > key
> > > > version. Existing
> > > > files do not have their EDEKs reencrypted using the new ezkeyid/
> > > > version, but this will be considered as a future enhancement. Note
> > that a
> > > > key rotation only needs to causes a reencryption of the DEK, not a
> > > > reencryption of the underlying file.
> > > >
> > >
> >
>

--001a113fefde6482e0051876409d--