hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yuren Wu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13173) SecurityUtil.buildtokenService broke token authentication in multi-home network env
Date Wed, 18 May 2016 18:54:13 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289588#comment-15289588
] 

Yuren Wu commented on HADOOP-13173:
-----------------------------------

Looking through the code and i think in this particular case I think the best place to setup
the value for useIp is in DFSUtil.createKeyProvider method  
This static method has visibility of job confiig and can easily setup the boolean value .
Given KMS is a generic service to provide keys, the best place to handle this is to take the
parameter into KMSClientProvider and cascading this flag until it reach the point to call
SecurityUtil. However, that will affect too many APIs. At this moment the KMS is used by HDFS
. therefore it can be done using this single fix.  

The fix code is in the attachment. 


> SecurityUtil.buildtokenService broke token authentication in multi-home network env
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-13173
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13173
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 2.6.0
>         Environment: multi-homed network cluster on SLES 11
>            Reporter: Yuren Wu
>
> Hadoop cluster: multihomed network with KMS and Kerberos. 
> Sympton: mapreduce mapper will fail with kerberos error. (stack trace later)
> Let's assume hadoop cluster has two networks, PRIVATE_NET and PUBLIC_NET. 
> When mapreduce job submitted to the cluster from an outside ndoe (node only has visibility
to PUBLIC_NET, it will acquire KMS tokens based on PUBLIC_NET ip address. Such token in Credential's
token list using IP will not be correct resolved in the cluster. Therefore, we are using hadoop.security.token.service.use_ip=false
in the client side core-site.xml file.  
> However, once job is accepted by yarn, the mapper container still throws kerberos error.
A detailed trace shows that when DFSClient is calling decryptEncryptedDataencryptionKey function,
it will reach to org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection
method. This method will call SecurityUtil.buildtokenService to obtain the service name and
use it to lookup the token in Credentials.getToken(service) call. This call returned the service
in the IP:PORT format. Therefore, no token can be retrieved from credential's token hashmap.

> After read SecurityUtil class, I think this class does not load configuration from configuration
file at all. It seems to me that this class create a default configuration object and the
useIpForTokenService variable is always set default value true.. 
> I do not have enough experience to provide a fix. In my mind, somewhere we need to provide
the correct value of useIpForTokenService based on job setup.
> Stack trace provided here. this is from wordcount job from example. 
> 2016-05-17 22:11:49,428 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running
child : java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException:
 GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos
tgt)
>         at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:503)
>         at org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:766)
>         at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$3.call(LoadBalancingKMSClientProvider.java:185)
>         at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$3.call(LoadBalancingKMSClientProvider.java:181)
>         at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:94)
>         at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:181)
>         at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388)
>         at org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1419)
>         at org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1489)
>         at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:310)
>         at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:304)
>         at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>         at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:304)
>         at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:775)
>         at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:85)
>         at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:548)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:786)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
>         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException:
GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos
tgt)
>         at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:318)
>         at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:203)
>         at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127)
>         at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
>         at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:373)
>         at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:497)
>         at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:493)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
>         at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:493)
>         ... 22 more
> Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)
>         at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
>         at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
>         at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
>         at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
>         at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
>         at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
>         at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:297)
>         at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:275)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>         at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:275)
>         ... 32 more



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message