hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yuren Wu (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-13173) SecurityUtil.buildtokenService broke token authentication in multi-home network env
Date Wed, 18 May 2016 14:49:12 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-13173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Yuren Wu updated HADOOP-13173:
------------------------------
    Description: 
Hadoop cluster: multihomed network with KMS and Kerberos. 
Sympton: mapreduce mapper will fail with kerberos error. (stack trace later)

Let's assume hadoop cluster has two networks, PRIVATE_NET and PUBLIC_NET. 
When mapreduce job submitted to the cluster from an outside ndoe (node only has visibility
to PUBLIC_NET, it will acquire KMS tokens based on PUBLIC_NET ip address. Such token in Credential's
token list using IP will not be correct resolved in the cluster. Therefore, we are using hadoop.security.token.service.use_ip=false
in the client side core-site.xml file.  
However, once job is accepted by yarn, the mapper container still throws kerberos error. A
detailed trace shows that when DFSClient is calling decryptEncryptedDataencryptionKey function,
it will reach to org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection
method. This method will call SecurityUtil.buildtokenService to obtain the service name and
use it to lookup the token in Credentials.getToken(service) call. This call returned the service
in the IP:PORT format. Therefore, no token can be retrieved from credential's token hashmap.


After read SecurityUtil class, I think this class does not load configuration from configuration
file at all. It seems to me that this class create a default configuration object and the
useIpForTokenService variable is always set default value true.. 

I do not have enough experience to provide a fix. In my mind, somewhere we need to provide
the correct value of useIpForTokenService based on job setup.


Stack trace provided here. this is from wordcount job from example. 
2016-05-17 22:11:49,428 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running
child : java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException:
 GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos
tgt)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:503)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:766)
        at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$3.call(LoadBalancingKMSClientProvider.java:185)
        at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$3.call(LoadBalancingKMSClientProvider.java:181)
        at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:94)
        at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:181)
        at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388)
        at org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1419)
        at org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1489)
        at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:310)
        at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:304)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:304)
        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:775)
        at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:85)
        at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:548)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:786)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException:
No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:318)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:203)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127)
        at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:373)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:497)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:493)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:493)
        ... 22 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any
Kerberos tgt)
        at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
        at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
        at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
        at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:297)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:275)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:275)
        ... 32 more


  was:
Hadoop cluster: multihomed network with KMS and Kerberos. 
Sympton: mapreduce mapper will fail with kerberos error. (stack trace later)

Let's assume hadoop cluster has two networks, PRIVATE_NET and PUBLIC_NET. 
When mapreduce job submitted to the cluster from an outside ndoe (node only has visibility
to PUBLIC_NET, it will acquire KMS tokens based on PUBLIC_NET ip address. Such token in Credential's
token list using IP will not be correct resolved in the cluster. Therefore, we are using hadoop.security.token.service.use_ip=false
in the client side core-site.xml file.  
However, once job is accepted by yarn, the mapper container still throws kerberos error. A
detailed trace shows that when DFSClient is calling decryptEncryptedDataencryptionKey function,
it will reach to org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection
method. This method will call SecurityUtil.buildtokenService to obtain the service name and
use it to lookup the token in Credentials.getToken(service) call. This call returned the service
in the IP:PORT format. Therefore, no token can be retrieved from credential's token hashmap.


After read SecurityUtil class, I think this class does not load configuration from configuration
file at all. It seems to me that this class create a default configuration object and the
useIpForTokenService variable is always set false. 

I do not have enough experience to provide a fix. In my mind, somewhere we need to provide
the correct value of useIpForTokenService based on job setup.


Stack trace provided here. this is from wordcount job from example. 
2016-05-17 22:11:49,428 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running
child : java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException:
 GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos
tgt)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:503)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:766)
        at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$3.call(LoadBalancingKMSClientProvider.java:185)
        at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$3.call(LoadBalancingKMSClientProvider.java:181)
        at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:94)
        at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:181)
        at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388)
        at org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1419)
        at org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1489)
        at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:310)
        at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:304)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:304)
        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:775)
        at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:85)
        at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:548)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:786)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException:
No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:318)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:203)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127)
        at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:373)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:497)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:493)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:493)
        ... 22 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any
Kerberos tgt)
        at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
        at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
        at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
        at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:297)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:275)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:275)
        ... 32 more



> SecurityUtil.buildtokenService broke token authentication in multi-home network env
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-13173
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13173
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 2.6.0
>         Environment: multi-homed network cluster on SLES 11
>            Reporter: Yuren Wu
>
> Hadoop cluster: multihomed network with KMS and Kerberos. 
> Sympton: mapreduce mapper will fail with kerberos error. (stack trace later)
> Let's assume hadoop cluster has two networks, PRIVATE_NET and PUBLIC_NET. 
> When mapreduce job submitted to the cluster from an outside ndoe (node only has visibility
to PUBLIC_NET, it will acquire KMS tokens based on PUBLIC_NET ip address. Such token in Credential's
token list using IP will not be correct resolved in the cluster. Therefore, we are using hadoop.security.token.service.use_ip=false
in the client side core-site.xml file.  
> However, once job is accepted by yarn, the mapper container still throws kerberos error.
A detailed trace shows that when DFSClient is calling decryptEncryptedDataencryptionKey function,
it will reach to org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection
method. This method will call SecurityUtil.buildtokenService to obtain the service name and
use it to lookup the token in Credentials.getToken(service) call. This call returned the service
in the IP:PORT format. Therefore, no token can be retrieved from credential's token hashmap.

> After read SecurityUtil class, I think this class does not load configuration from configuration
file at all. It seems to me that this class create a default configuration object and the
useIpForTokenService variable is always set default value true.. 
> I do not have enough experience to provide a fix. In my mind, somewhere we need to provide
the correct value of useIpForTokenService based on job setup.
> Stack trace provided here. this is from wordcount job from example. 
> 2016-05-17 22:11:49,428 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running
child : java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException:
 GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos
tgt)
>         at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:503)
>         at org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:766)
>         at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$3.call(LoadBalancingKMSClientProvider.java:185)
>         at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$3.call(LoadBalancingKMSClientProvider.java:181)
>         at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:94)
>         at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:181)
>         at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388)
>         at org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1419)
>         at org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1489)
>         at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:310)
>         at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:304)
>         at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>         at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:304)
>         at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:775)
>         at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:85)
>         at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:548)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:786)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
>         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException:
GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos
tgt)
>         at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:318)
>         at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:203)
>         at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127)
>         at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
>         at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:373)
>         at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:497)
>         at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:493)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
>         at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:493)
>         ... 22 more
> Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)
>         at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
>         at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
>         at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
>         at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
>         at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
>         at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
>         at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:297)
>         at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:275)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>         at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:275)
>         ... 32 more



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message