hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yuren Wu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6565) Configuration to use host name in delegation token service is not read from job.xml during MapReduce job execution.
Date Wed, 23 Nov 2016 23:26:58 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15691638#comment-15691638
] 

Yuren Wu commented on MAPREDUCE-6565:
-------------------------------------

Thanks for all the attention you guys have paid to this issue. My initial hack works well
with MR job, however when I did a test on Spark job submitted remotely to yarn cluster, the
hack did not work. i did not open another jira for spark job because i did not quite dig into
the spark job settings to see who it was handled. Maybe you guys have more insight into the
code and can figure out how to solve this problem for all YARN managed applications. 


> Configuration to use host name in delegation token service is not read from job.xml during
MapReduce job execution.
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6565
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6565
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Chris Nauroth
>            Assignee: Li Lu
>         Attachments: MAPREDUCE-6565-trunk.001.patch
>
>
> By default, the service field of a delegation token is populated based on server IP address.
 Setting {{hadoop.security.token.service.use_ip}} to {{false}} changes this behavior to use
host name instead of IP address.  However, this configuration property is not read from job.xml.
 Instead, it's read from a separate {{Configuration}} instance created during static initialization
of {{SecurityUtil}}.  This does not work correctly with MapReduce jobs if the framework is
distributed by setting {{mapreduce.application.framework.path}} and the {{mapreduce.application.classpath}}
is isolated to avoid reading core-site.xml from the cluster nodes.  MapReduce tasks will fail
to authenticate to HDFS, because they'll try to find a delegation token based on the NameNode
IP address, even though at job submission time the tokens were generated using the host name.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message