hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Abdelnur (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-7733) Mapreduce jobs are failing when JT has hadoop.security.token.service.use_ip=false and client has hadoop.security.token.service.use_ip=true
Date Thu, 01 Dec 2011 17:54:40 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161004#comment-13161004
] 

Alejandro Abdelnur commented on HADOOP-7733:
--------------------------------------------

In some Oozie setups, I'm seeing this exception at tgt renewal time. This means, the original
login is successful and access to HDFS is OK, but when renewal time comes this exception pops
in the logs and HDFS access is no more. In other words, same same but different.

Wondering if this could be related to this misconfiguration.

                
> Mapreduce jobs are failing when JT has hadoop.security.token.service.use_ip=false and
client has hadoop.security.token.service.use_ip=true
> ------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7733
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7733
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.20.205.0
>            Reporter: Rajit Saha
>            Assignee: Daryn Sharp
>
> I have added following property in core-site.xml of all the nodes in cluster and restarted
> <property>
> <name>hadoop.security.token.service.use_ip</name>
> <value>false</value>
> <description>desc</description>
> <final></final>
> </property>
> Then ran a randomwriter, distcp jobs, they are all failing 
> $HADOOP_HOME/bin/hadoop --config $HADOOP_CONFIG_DIR jar $HADOOP_HOME/hadoop-examples.jar
randomwriter -Dtest.randomwrite.bytes_per_map=256000 input_1318325953
> Running 140 maps.
> Job started: Tue Oct 11 09:48:09 UTC 2011
> 11/10/11 09:48:09 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 14 for <USERNAME>
on <Namenode IP>:8020
> 11/10/11 09:48:09 INFO security.TokenCache: Got dt for
> hdfs://<Namenode Hostname>/user/<USERNAME>/.staging/job_201110110946_0001;uri=<Namenode
IP>:8020;t.service=<Namenode IP>:8020
> 11/10/11 09:48:09 INFO mapred.JobClient: Cleaning up the staging area
> hdfs://<Namenode Hostname>/user/<USERNAME>/.staging/job_201110110946_0001
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.io.IOException: Call
to
> <Namenode Hostname>/<Namenode IP>:8020 failed on local exception: java.io.IOException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid
credentials provided
> (Mechanism level: Failed to find any Kerberos tgt)]
>         at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3943)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
> Caused by: java.io.IOException: Call to <Namenode Hostname>/<Namenode IP>:8020
failed on local exception:
> java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid
> credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>         at org.apache.hadoop.ipc.Client.wrapException(Client.java:1103)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1071)
>         at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
>         at $Proxy7.getProtocolVersion(Unknown Source)
>         at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
>         at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
>         at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:118)
>         at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:222)
>         at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:187)
>         at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
>         at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1328)
>         at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:65)
>         at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1346)
>         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:244)
>         at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
>         at org.apache.hadoop.mapred.JobInProgress$2.run(JobInProgress.java:401)
>         at org.apache.hadoop.mapred.JobInProgress$2.run(JobInProgress.java:399)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
>         at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:399)
>         at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3941)
>         ... 11 more
> Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: No
> valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>         at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:539)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
>         at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:484)
>         at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:586)
>         at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184)
>         at org.apache.hadoop.ipc.Client.getConnection(Client.java:1202)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1046)
>         ... 31 more
> Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials
> provided (Mechanism level: Failed to find any Kerberos tgt)]
>         at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
>         at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:134)
>         at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:381)
>         at org.apache.hadoop.ipc.Client$Connection.access$1100(Client.java:184)
>         at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:579)
>         at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:576)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
>         at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:575)
>         ... 34 more
> Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)
>         at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:130)
>         at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:106)
>         at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:172)
>         at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:209)
>         at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195)
>         at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162)
>         at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175)
>         ... 43 more
>         at org.apache.hadoop.ipc.Client.call(Client.java:1066)
>         at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
>         at org.apache.hadoop.mapred.$Proxy7.submitJob(Unknown Source)
>         at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:913)
>         at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:842)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
>         at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:842)
>         at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:816)
>         at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1253)
>         at org.apache.hadoop.examples.RandomWriter.run(RandomWriter.java:272)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>         at org.apache.hadoop.examples.RandomWriter.main(RandomWriter.java:283)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
>         at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
>         at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> At the sametime Jobtracker log says
> ===================================
> 2011-10-11 09:48:09,486 WARN org.apache.hadoop.ipc.Client: Exception encountered while
connecting to the server :
> javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid
credentials provided
> (Mechanism level: Failed to find any Kerberos tgt)]
> 2011-10-11 09:48:09,487 INFO org.apache.hadoop.ipc.Server: IPC Server handler 26 on 50300,
call
> submitJob(job_201110110946_0001, hdfs://<Namenode hostname> /user/<USER>/.staging/job_201110110946_0001,
> org.apache.hadoop.security.Credentials@16381a53) from <client IP> :46859: error:
java.io.IOException:
> java.io.IOException: Call to <Namenode hostname> /<Namenode IP> :8020 failed
on local exception:
> java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid
> credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
> java.io.IOException: java.io.IOException: Call to <Namenode hostname> /<Namenode
IP> :8020 failed on local
> exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: No
> valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>         at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3943)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
> Caused by: java.io.IOException: Call to <Namenode hostname> /<Namenode IP>
:8020 failed on local exception:
> java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid
> credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>         at org.apache.hadoop.ipc.Client.wrapException(Client.java:1103)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1071)
>         at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
>         at $Proxy7.getProtocolVersion(Unknown Source)
>         at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
>         at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
>         at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:118)
>         at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:222)
>         at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:187)
>         at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
>         at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1328)
>         at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:65)
>         at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1346)
>         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:244)
>         at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
>         at org.apache.hadoop.mapred.JobInProgress$2.run(JobInProgress.java:401)
>         at org.apache.hadoop.mapred.JobInProgress$2.run(JobInProgress.java:399)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
>         at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:399)
>         at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3941)
>         ... 11 more
> Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: No
> valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>         at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:539)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
>         at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:484)
>         at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:586)
>         at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184)
>         at org.apache.hadoop.ipc.Client.getConnection(Client.java:1202)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1046)
>         ... 31 more
> Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials
> provided (Mechanism level: Failed to find any Kerberos tgt)]
>         at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
>         at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:134)
>         at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:381)
>         at org.apache.hadoop.ipc.Client$Connection.access$1100(Client.java:184)
>         at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:579)
>         at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:576)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
>         at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:575)
>         ... 34 more
> Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)
>         at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:130)
>         at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:106)
>         at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:172)
>         at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:209)
>         at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195)
>         at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162)
>         at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175)
>         ... 43 more

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message