Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Fri, 16 Mar 2012 00:24:38 +0000 (UTC)
From: "Hudson (Commented) (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: 
 <1754174104.22222.1331857478863.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HDFS-3062) Fail to submit mapred job on a
 secured-HA-HDFS: logic URI cannot be picked up by job submission.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-3062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13230749#comment-13230749 ] 

Hudson commented on HDFS-3062:
------------------------------

Integrated in Hadoop-Common-0.23-Commit #687 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/687/])
    HDFS-3062. Fix bug which prevented MR job submission from creating delegation tokens on an HA cluster. Contributed by Mingjie Lai. (Revision 1301286)

     Result = SUCCESS
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301286
Files : 
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java

                
> Fail to submit mapred job on a secured-HA-HDFS: logic URI cannot be picked up by job submission.
> ------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-3062
>                 URL: https://issues.apache.org/jira/browse/HDFS-3062
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: ha, security
>    Affects Versions: 0.24.0
>            Reporter: Mingjie Lai
>            Assignee: Mingjie Lai
>            Priority: Critical
>             Fix For: 0.24.0, 0.23.3
>
>         Attachments: HDFS-3062-trunk-2.patch, HDFS-3062-trunk.patch
>
>
> When testing the combination of NN HA + security + yarn, I found that the mapred job submission cannot pick up the logic URI of a nameservice. 
> I have logic URI configured in core-site.xml
> {code}
> <property>
>  <name>fs.defaultFS</name>
>  <value>hdfs://ns1</value>
> </property>
> {code}
> HDFS client can work with the HA deployment/configs:
> {code}
> [root@nn1 hadoop]# hdfs dfs -ls /
> Found 6 items
> drwxr-xr-x   - hbase  hadoop          0 2012-03-07 20:42 /hbase
> drwxrwxrwx   - yarn   hadoop          0 2012-03-07 20:42 /logs
> drwxr-xr-x   - mapred hadoop          0 2012-03-07 20:42 /mapred
> drwxr-xr-x   - mapred hadoop          0 2012-03-07 20:42 /mr-history
> drwxrwxrwt   - hdfs   hadoop          0 2012-03-07 21:57 /tmp
> drwxr-xr-x   - hdfs   hadoop          0 2012-03-07 20:42 /user
> {code}
> but cannot submit a mapred job with security turned on
> {code}
> [root@nn1 hadoop]# /usr/lib/hadoop/bin/yarn --config ./conf jar share/hadoop/mapreduce/hadoop-mapreduce-examples-0.24.0-SNAPSHOT.jar randomwriter out
> Running 0 maps.
> Job started: Wed Mar 07 23:28:23 UTC 2012
> java.lang.IllegalArgumentException: java.net.UnknownHostException: ns1
> 	at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:431)
> 	at org.apache.hadoop.security.SecurityUtil.buildDTServiceName(SecurityUtil.java:312)
> 	at org.apache.hadoop.fs.FileSystem.getCanonicalServiceName(FileSystem.java:217)
> 	at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:119)
> 	at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:97)
> 	at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80)
> 	at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:137)
> 	at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:411)
> 	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:326)
> 	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1221)
> 	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
> ....
> {code}0.24

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira