Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-dev@hadoop.apache.org
Date: Thu, 8 Mar 2012 01:01:09 +0000 (UTC)
From: "Mingjie Lai (Created) (JIRA)" <jira@apache.org>
To: common-dev@hadoop.apache.org
Message-ID: 
 <672323370.37211.1331168469457.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-8153) Fail to submit mapred job on a
 secured-HA-HDFS: logic URI cannot be picked up by job submission.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Fail to submit mapred job on a secured-HA-HDFS: logic URI cannot be picked up by job submission.
------------------------------------------------------------------------------------------------

                 Key: HADOOP-8153
                 URL: https://issues.apache.org/jira/browse/HADOOP-8153
             Project: Hadoop Common
          Issue Type: Bug
          Components: ha, security
    Affects Versions: 0.24.0
            Reporter: Mingjie Lai
             Fix For: 0.24.0


When testing the combination of NN HA + security + yarn, I found that the mapred job submission cannot pick up the logic URI of a nameservice. 

I have logic URI configured in core-site.xml
{code}
<property>
 <name>fs.defaultFS</name>
 <value>hdfs://ns1</value>
</property>
{code}

HDFS client can work with the HA deployment/configs:
{code}
[root@nn1 hadoop]# hdfs dfs -ls /
Found 6 items
drwxr-xr-x   - hbase  hadoop          0 2012-03-07 20:42 /hbase
drwxrwxrwx   - yarn   hadoop          0 2012-03-07 20:42 /logs
drwxr-xr-x   - mapred hadoop          0 2012-03-07 20:42 /mapred
drwxr-xr-x   - mapred hadoop          0 2012-03-07 20:42 /mr-history
drwxrwxrwt   - hdfs   hadoop          0 2012-03-07 21:57 /tmp
drwxr-xr-x   - hdfs   hadoop          0 2012-03-07 20:42 /user
{code}

but cannot submit a mapred job with security turned on
{code}
[root@nn1 hadoop]# /usr/lib/hadoop/bin/yarn --config ./conf jar share/hadoop/mapreduce/hadoop-mapreduce-examples-0.24.0-SNAPSHOT.jar randomwriter out
Running 0 maps.
Job started: Wed Mar 07 23:28:23 UTC 2012
java.lang.IllegalArgumentException: java.net.UnknownHostException: ns1
	at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:431)
	at org.apache.hadoop.security.SecurityUtil.buildDTServiceName(SecurityUtil.java:312)
	at org.apache.hadoop.fs.FileSystem.getCanonicalServiceName(FileSystem.java:217)
	at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:119)
	at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:97)
	at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80)
	at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:137)
	at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:411)
	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:326)
	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1221)
	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
....
{code}0.24

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira