Return-Path: 
 <common-issues-return-21380-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9E1419E00
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>;
 Tue, 11 Oct 2011 10:29:40 +0000 (UTC)
Received: (qmail 95085 invoked by uid 500); 11 Oct 2011 10:29:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94610 invoked by uid 500); 11 Oct 2011 10:29:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94593 invoked by uid 99); 11 Oct 2011 10:29:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Oct 2011 10:29:34 +0000
X-ASF-Spam-Status: No, hits=-2000.5 required=5.0
	tests=ALL_TRUSTED,RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Oct 2011 10:29:32 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DE11636267
	for <common-issues@hadoop.apache.org>; Tue, 11 Oct 2011 10:29:11 +0000 (UTC)
Date: Tue, 11 Oct 2011 10:29:11 +0000 (UTC)
From: "Rajit Saha (Created) (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: 
 <1663016577.173.1318328951910.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7733) Mapreduce jobs are failing with
 hadoop.security.token.service.use_ip=false
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Mapreduce jobs are failing with hadoop.security.token.service.use_ip=false
--------------------------------------------------------------------------

                 Key: HADOOP-7733
                 URL: https://issues.apache.org/jira/browse/HADOOP-7733
             Project: Hadoop Common
          Issue Type: Bug
          Components: security
    Affects Versions: 0.20.205.0
            Reporter: Rajit Saha


I have added following property in core-site.xml of all the nodes in cluster and restarted

<property>
<name>hadoop.security.token.service.use_ip</name>
<value>false</value>
<description>desc</description>
<final></final>
</property>

Then ran a randomwriter, distcp jobs, they are all failing 
$HADOOP_HOME/bin/hadoop --config $HADOOP_CONFIG_DIR jar $HADOOP_HOME/hadoop-examples.jar randomwriter -Dtest.randomwrite.bytes_per_map=256000 input_1318325953
Running 140 maps.
Job started: Tue Oct 11 09:48:09 UTC 2011
11/10/11 09:48:09 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 14 for <USERNAME> on <Namenode IP>:8020
11/10/11 09:48:09 INFO security.TokenCache: Got dt for
hdfs://<Namenode Hostname>/user/<USERNAME>/.staging/job_201110110946_0001;uri=<Namenode IP>:8020;t.service=<Namenode IP>:8020
11/10/11 09:48:09 INFO mapred.JobClient: Cleaning up the staging area
hdfs://<Namenode Hostname>/user/<USERNAME>/.staging/job_201110110946_0001
org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.io.IOException: Call to
<Namenode Hostname>/<Namenode IP>:8020 failed on local exception: java.io.IOException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos tgt)]
        at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3943)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
Caused by: java.io.IOException: Call to <Namenode Hostname>/<Namenode IP>:8020 failed on local exception:
java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid
credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
        at org.apache.hadoop.ipc.Client.wrapException(Client.java:1103)
        at org.apache.hadoop.ipc.Client.call(Client.java:1071)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
        at $Proxy7.getProtocolVersion(Unknown Source)
        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
        at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:118)
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:222)
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:187)
        at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1328)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:65)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1346)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:244)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
        at org.apache.hadoop.mapred.JobInProgress$2.run(JobInProgress.java:401)
        at org.apache.hadoop.mapred.JobInProgress$2.run(JobInProgress.java:399)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
        at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:399)
        at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3941)
        ... 11 more
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No
valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
        at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:539)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
        at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:484)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:586)
        at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1202)
        at org.apache.hadoop.ipc.Client.call(Client.java:1046)
        ... 31 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials
provided (Mechanism level: Failed to find any Kerberos tgt)]
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
        at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:134)
        at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:381)
        at org.apache.hadoop.ipc.Client$Connection.access$1100(Client.java:184)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:579)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:576)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:575)
        ... 34 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
        at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:130)
        at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:106)
        at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:172)
        at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:209)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162)
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175)
        ... 43 more

        at org.apache.hadoop.ipc.Client.call(Client.java:1066)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
        at org.apache.hadoop.mapred.$Proxy7.submitJob(Unknown Source)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:913)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:842)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:842)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:816)
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1253)
        at org.apache.hadoop.examples.RandomWriter.run(RandomWriter.java:272)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.examples.RandomWriter.main(RandomWriter.java:283)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
        at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
        at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)



At the sametime Jobtracker log says
===================================
2011-10-11 09:48:09,486 WARN org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server :
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos tgt)]
2011-10-11 09:48:09,487 INFO org.apache.hadoop.ipc.Server: IPC Server handler 26 on 50300, call
submitJob(job_201110110946_0001, hdfs://<Namenode hostname> /user/<USER>/.staging/job_201110110946_0001,
org.apache.hadoop.security.Credentials@16381a53) from <client IP> :46859: error: java.io.IOException:
java.io.IOException: Call to <Namenode hostname> /<Namenode IP> :8020 failed on local exception:
java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid
credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
java.io.IOException: java.io.IOException: Call to <Namenode hostname> /<Namenode IP> :8020 failed on local
exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No
valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
        at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3943)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
Caused by: java.io.IOException: Call to <Namenode hostname> /<Namenode IP> :8020 failed on local exception:
java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid
credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
        at org.apache.hadoop.ipc.Client.wrapException(Client.java:1103)
        at org.apache.hadoop.ipc.Client.call(Client.java:1071)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
        at $Proxy7.getProtocolVersion(Unknown Source)
        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
        at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:118)
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:222)
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:187)
        at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1328)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:65)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1346)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:244)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
        at org.apache.hadoop.mapred.JobInProgress$2.run(JobInProgress.java:401)
        at org.apache.hadoop.mapred.JobInProgress$2.run(JobInProgress.java:399)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
        at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:399)
        at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3941)
        ... 11 more

Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No
valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
        at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:539)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
        at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:484)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:586)
        at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1202)
        at org.apache.hadoop.ipc.Client.call(Client.java:1046)
        ... 31 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials
provided (Mechanism level: Failed to find any Kerberos tgt)]
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
        at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:134)
        at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:381)
        at org.apache.hadoop.ipc.Client$Connection.access$1100(Client.java:184)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:579)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:576)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:575)
        ... 34 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
        at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:130)
        at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:106)
        at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:172)
        at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:209)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162)
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175)
        ... 43 more


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira