hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11393) Hadoop KMS contacted by jobs which don’t use KMS encryption
Date Tue, 07 Feb 2017 15:00:47 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856158#comment-15856158

Daryn Sharp commented on HDFS-11393:

We've seen the same in our initial testing.  There's not a good way to selectively obtain
the token other than add/remove the kms conf setting.  That's certainly less than ideal  since
EZs are supposed to be transparent to the user.

One impediment with the current model is that apis for obtaining tokens aren't path based
but filesystem based.  If they were path based and clients correctly premeditated all paths
to be accessed, the token could be obtained if needed.  But...  Even if a client premeditates
it will access a given path, there's no way to know that deeper in that path is an EZ.  Symlinks
pose a similar issue.

About the only real way to solve the problem (in the current security model) is allowing the
client to auth to the kms via the NN token.  That would be a massive change that creates a
tight coupling with the services, requires shared key distribution, more critical path moving
parts, complicates the kms ability to enforce its own ACLs, etc.

In the end, for us the job submission rate is so insignificant compared to all other job generated
load that it's an acceptable, albeit unnecessary, overhead.

> Hadoop KMS contacted by jobs which don’t use  KMS encryption
> ------------------------------------------------------------
>                 Key: HDFS-11393
>                 URL: https://issues.apache.org/jira/browse/HDFS-11393
>             Project: Hadoop HDFS
>          Issue Type: Wish
>         Environment: Hadoop 2.7.3, Spark 1.6.3 on Yarn, Oozie 4.2.3
> Cluster secured with Kerberos
>            Reporter: Alexandre Linte
>            Priority: Minor
> Hello,
> After few days of usage of Hadoop KMS in our pre-production platform, it was noticed
that after restarting resourcemanagers, all Yarn jobs generated on the platform interrogated
the KMS server, even if the didn't process encrypted information. 
> {noformat}
> 2016-11-23 10:58:47,708 DEBUG AuthenticationFilter - Request [http://uabigkms01:16000/kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fuabigrm01%40SANDBOX.HADOOP]
triggering authentication
> 2016-11-23 10:58:47,735 DEBUG AuthenticationFilter - Request [http://uabigkms01:16000/kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fuabigrm01%40SANDBOX.HADOOP]
user xxxx authenticated
> {noformat}
> Indeed after research we see that KMS supports delegation token to authenticate to the
Java KeyProvider by processes without Kerberos credentials.
> Is there a way to bypass Delegation Token on KMS and just contact KMS when jobs or user
into HDFS use encrypted data ?

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message