Return-Path: X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7D2E09E74 for ; Tue, 11 Oct 2011 15:25:35 +0000 (UTC) Received: (qmail 89509 invoked by uid 500); 11 Oct 2011 15:25:35 -0000 Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org Received: (qmail 89478 invoked by uid 500); 11 Oct 2011 15:25:34 -0000 Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-issues@hadoop.apache.org Delivered-To: mailing list common-issues@hadoop.apache.org Received: (qmail 89470 invoked by uid 99); 11 Oct 2011 15:25:34 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Oct 2011 15:25:34 +0000 X-ASF-Spam-Status: No, hits=-2000.5 required=5.0 tests=ALL_TRUSTED,RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Oct 2011 15:25:32 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116]) by hel.zones.apache.org (Postfix) with ESMTP id 2102C303937 for ; Tue, 11 Oct 2011 15:25:12 +0000 (UTC) Date: Tue, 11 Oct 2011 15:25:12 +0000 (UTC) From: "Daryn Sharp (Assigned) (JIRA)" To: common-issues@hadoop.apache.org Message-ID: <454458600.1125.1318346712136.JavaMail.tomcat@hel.zones.apache.org> In-Reply-To: <1663016577.173.1318328951910.JavaMail.tomcat@hel.zones.apache.org> Subject: [jira] [Assigned] (HADOOP-7733) Mapreduce jobs are failing with hadoop.security.token.service.use_ip=false MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HADOOP-7733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp reassigned HADOOP-7733: ----------------------------------- Assignee: Daryn Sharp > Mapreduce jobs are failing with hadoop.security.token.service.use_ip=false > -------------------------------------------------------------------------- > > Key: HADOOP-7733 > URL: https://issues.apache.org/jira/browse/HADOOP-7733 > Project: Hadoop Common > Issue Type: Bug > Components: security > Affects Versions: 0.20.205.0 > Reporter: Rajit Saha > Assignee: Daryn Sharp > > I have added following property in core-site.xml of all the nodes in cluster and restarted > > hadoop.security.token.service.use_ip > false > desc > > > Then ran a randomwriter, distcp jobs, they are all failing > $HADOOP_HOME/bin/hadoop --config $HADOOP_CONFIG_DIR jar $HADOOP_HOME/hadoop-examples.jar randomwriter -Dtest.randomwrite.bytes_per_map=256000 input_1318325953 > Running 140 maps. > Job started: Tue Oct 11 09:48:09 UTC 2011 > 11/10/11 09:48:09 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 14 for on :8020 > 11/10/11 09:48:09 INFO security.TokenCache: Got dt for > hdfs:///user//.staging/job_201110110946_0001;uri=:8020;t.service=:8020 > 11/10/11 09:48:09 INFO mapred.JobClient: Cleaning up the staging area > hdfs:///user//.staging/job_201110110946_0001 > org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.io.IOException: Call to > /:8020 failed on local exception: java.io.IOException: > javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided > (Mechanism level: Failed to find any Kerberos tgt)] > at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3943) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382) > Caused by: java.io.IOException: Call to /:8020 failed on local exception: > java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid > credentials provided (Mechanism level: Failed to find any Kerberos tgt)] > at org.apache.hadoop.ipc.Client.wrapException(Client.java:1103) > at org.apache.hadoop.ipc.Client.call(Client.java:1071) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225) > at $Proxy7.getProtocolVersion(Unknown Source) > at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396) > at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379) > at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:118) > at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:222) > at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:187) > at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89) > at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1328) > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:65) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1346) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:244) > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187) > at org.apache.hadoop.mapred.JobInProgress$2.run(JobInProgress.java:401) > at org.apache.hadoop.mapred.JobInProgress$2.run(JobInProgress.java:399) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) > at org.apache.hadoop.mapred.JobInProgress.(JobInProgress.java:399) > at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3941) > ... 11 more > Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No > valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] > at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:539) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) > at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:484) > at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:586) > at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184) > at org.apache.hadoop.ipc.Client.getConnection(Client.java:1202) > at org.apache.hadoop.ipc.Client.call(Client.java:1046) > ... 31 more > Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials > provided (Mechanism level: Failed to find any Kerberos tgt)] > at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194) > at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:134) > at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:381) > at org.apache.hadoop.ipc.Client$Connection.access$1100(Client.java:184) > at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:579) > at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:576) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) > at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:575) > ... 34 more > Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) > at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:130) > at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:106) > at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:172) > at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:209) > at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195) > at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162) > at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175) > ... 43 more > at org.apache.hadoop.ipc.Client.call(Client.java:1066) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225) > at org.apache.hadoop.mapred.$Proxy7.submitJob(Unknown Source) > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:913) > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:842) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) > at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:842) > at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:816) > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1253) > at org.apache.hadoop.examples.RandomWriter.run(RandomWriter.java:272) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at org.apache.hadoop.examples.RandomWriter.main(RandomWriter.java:283) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) > at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) > at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:156) > At the sametime Jobtracker log says > =================================== > 2011-10-11 09:48:09,486 WARN org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : > javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided > (Mechanism level: Failed to find any Kerberos tgt)] > 2011-10-11 09:48:09,487 INFO org.apache.hadoop.ipc.Server: IPC Server handler 26 on 50300, call > submitJob(job_201110110946_0001, hdfs:// /user//.staging/job_201110110946_0001, > org.apache.hadoop.security.Credentials@16381a53) from :46859: error: java.io.IOException: > java.io.IOException: Call to / :8020 failed on local exception: > java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid > credentials provided (Mechanism level: Failed to find any Kerberos tgt)] > java.io.IOException: java.io.IOException: Call to / :8020 failed on local > exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No > valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] > at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3943) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382) > Caused by: java.io.IOException: Call to / :8020 failed on local exception: > java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid > credentials provided (Mechanism level: Failed to find any Kerberos tgt)] > at org.apache.hadoop.ipc.Client.wrapException(Client.java:1103) > at org.apache.hadoop.ipc.Client.call(Client.java:1071) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225) > at $Proxy7.getProtocolVersion(Unknown Source) > at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396) > at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379) > at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:118) > at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:222) > at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:187) > at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89) > at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1328) > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:65) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1346) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:244) > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187) > at org.apache.hadoop.mapred.JobInProgress$2.run(JobInProgress.java:401) > at org.apache.hadoop.mapred.JobInProgress$2.run(JobInProgress.java:399) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) > at org.apache.hadoop.mapred.JobInProgress.(JobInProgress.java:399) > at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3941) > ... 11 more > Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No > valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] > at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:539) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) > at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:484) > at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:586) > at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184) > at org.apache.hadoop.ipc.Client.getConnection(Client.java:1202) > at org.apache.hadoop.ipc.Client.call(Client.java:1046) > ... 31 more > Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials > provided (Mechanism level: Failed to find any Kerberos tgt)] > at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194) > at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:134) > at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:381) > at org.apache.hadoop.ipc.Client$Connection.access$1100(Client.java:184) > at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:579) > at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:576) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) > at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:575) > ... 34 more > Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) > at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:130) > at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:106) > at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:172) > at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:209) > at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195) > at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162) > at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175) > ... 43 more -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira