hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Wang <andrew.w...@cloudera.com>
Subject Re: Key Rotation in Data-at-Rest Encryption
Date Mon, 15 Jun 2015 21:56:48 GMT
Hi Sitaraman,

A key name can have multiple versions. When you roll a key (via its name),
a new version is created. When you fetch a key via name, you get the
current version. You can also explicitly fetch a particular key version.

I think what you term a "key alias" is the key name.

Regarding FEInfo conversion between a PB and xattr, see these lines:

https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java#L2993
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java#L1174

Hope this helps,
Andrew

On Sun, Jun 14, 2015 at 5:48 AM, Sitaraman Vilayannur <
vrsitaramanietflists@gmail.com> wrote:

> Hi Arun,
>  FileEncryptionInfo has both a getKeyName and a getKeyVersionName.  What
> distinguishes the concept of keyname and key version.
> It appears to me that the keyname is closer to key alias than a key
> version.  What is key version? Thanks much.
> Sitaraman
>
> On Sun, Jun 14, 2015 at 2:07 PM, Sitaraman Vilayannur <
> vrsitaramanietflists@gmail.com> wrote:
>
> > Hi Arun,
> >  Thanks for your patience. I have a related question In my application i
> > need to encrypt/decrypt files
> > from the map reduce phase and i need to support key rotation.  Can i
> > access the KMS from the map/reduce
> > phase to retrieve the key material from the key alias which i retrieve
> > from the FileEncryptionInfo class stored in the
> >  extended attribute of the file(for decryption)?
> >  Any pointers to how i can plug in various key providers such as java
> > keystore would be appreciated.
> > Is the toString method used to store the FileEncryptionInfo into the
> > extended attribute if so how is the
> > FileEncryptionInfo object retrieved back from the String.
> > Thanks again for your help.
> > Sitaraman
> >
> > On Sun, Jun 14, 2015 at 12:53 PM, Arun Suresh <asuresh@apache.org>
> wrote:
> >
> >> Apologize if I wasn't clear
> >>
> >> > Is the EZ key version same as an alias for the key?
> >> yup
> >>
> >> > the EDEK along with the EZ key version is stored in the FIleInfo
> >> FileInfo contains both EDEK and EZ key version. The FileInfo (you can
> look
> >> at the *org.apache.hadoop.fs.FileEncryptionInfo* class for more info)
> >> object is stored as the value of the extended attribute of that file.
> >>
> >> > How is the KeyMaterial derived from the KeyAlias and where is the
> >> mapping between
> >> the two stored? Is it in the KMS?
> >> Yup. KMS extends the *org.apache.hadoop.crypto.key.KeyProvider* class.
> You
> >> can take a look at it or a concrete implementation such as
> >> JavaKeyStoreProvider for more information.
> >>
> >> Also, you should probably direct questions related to HDFS encryption to
> >> hdfs-dev@hadoop.apache.org
> >>
> >> Cheers
> >> -Arun
> >>
> >>
> >>
> >> On Sun, Jun 14, 2015 at 12:11 AM, Sitaraman Vilayannur <
> >> vrsitaramanietflists@gmail.com> wrote:
> >>
> >> > Hi Arun,
> >> > Thanks for your response.
> >> > Could you explain this a bit further for me....
> >> > Is the EZ key version same as an alias for the key?
> >> > The EDEK is stored in the extended attributes of the file and the
> EZkey
> >> > Version is stored
> >> >  in the FileInfo  why is the EZKey Version not stored in the extended
> >> > attributes too.
> >> > Where is the FileInfo object persisted? Is it in the NameNode?
> >> > How is the KeyMaterial derived from the KeyAlias and where is the
> >> mapping
> >> > between the two stored? Is it in the KMS?
> >> > Thanks much for your help in this.
> >> > Sitaraman
> >> >
> >> > On Sun, Jun 14, 2015 at 12:14 PM, Arun Suresh <asuresh@cloudera.com>
> >> > wrote:
> >> >
> >> > > Hello Sitaraman,
> >> > >
> >> > > It is the EZ key "version" that is used to generate the EDEK (and
> >> which
> >> > is
> >> > > ultimately stored in the encrypted file's extended attributes
> >> > > '*raw.hdfs.crypto.encryption.info
> >> > > <http://raw.hdfs.crypto.encryption.info>*'), not really the
the EZ
> >> key
> >> > > itself (which is stored in the directory's extended attribute ‘
> >> > > *raw.hdfs.crypto.encryption.zone*’).
> >> > >
> >> > > Essentially, each file in a directory has a unique EDEK.. and an
> EDEK
> >> is
> >> > is
> >> > > generated with the current version of the directory EZ key. The EDEK
> >> > along
> >> > > with the EZ key version is stored in the FIleInfo. While decrypting,
> >> both
> >> > > these are passed on to the KMS which provides the client with the
> DEK
> >> > which
> >> > > can be used to decrypt the file.
> >> > >
> >> > > Hope this clarifies things.
> >> > >
> >> > > Cheers
> >> > > -Arun
> >> > >
> >> > > On Sat, Jun 13, 2015 at 9:51 PM, Sitaraman Vilayannur <
> >> > > vrsitaramanietflists@gmail.com> wrote:
> >> > >
> >> > > > HDFSDataatRestEncryption.pdf says the following about key
> >> > > rotation..(please
> >> > > > see appended below at the end of the mail)
> >> > > > If the existing files do not have their EDEKs reencrypted using
> the
> >> new
> >> > > > ezkeyid, how would the existing files be decrypted? That is where
> is
> >> > the
> >> > > > mapping between files and its EZKey (for after key rotation
> >> different
> >> > > files
> >> > > > have different EZKeys)ids stored and how is it retrieved?
> >> > > > Thanks
> >> > > > Sitaraman
> >> > > >
> >> > > > Key Rotation
> >> > > > When the administrator causes a key rotation of the EZkey
> >> > > > in the KMS, the encryption zone’s EZkey
> >> > > > (stored in the encryption zone directory’s
> >> > > raw.hdfs.crypto.encryption.zone
> >> > > > extended attribute) gets the new keyid and version (only the
> version
> >> > > > changes). Any new files
> >> > > > created in the encryption zone have their DEKs encrypted using
the
> >> new
> >> > > key
> >> > > > version. Existing
> >> > > > files do not have their EDEKs reencrypted using the new ezkeyid/
> >> > > > version, but this will be considered as a future enhancement.
Note
> >> > that a
> >> > > > key rotation only needs to causes a reencryption of the DEK,
not a
> >> > > > reencryption of the underlying file.
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message