hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varun Vasudev (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (YARN-2426) ResourceManger is not able renew WebHDFS token when application submitted by Yarn WebService
Date Mon, 08 Dec 2014 11:58:13 GMT

     [ https://issues.apache.org/jira/browse/YARN-2426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Varun Vasudev resolved YARN-2426.
---------------------------------
       Resolution: Fixed
    Fix Version/s: 2.6.0

Fixed with HDFS-6904 exposing an API to allow clients to set the service.

> ResourceManger is not able renew WebHDFS token when application submitted by Yarn WebService
> --------------------------------------------------------------------------------------------
>
>                 Key: YARN-2426
>                 URL: https://issues.apache.org/jira/browse/YARN-2426
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager, resourcemanager, webapp
>    Affects Versions: 2.6.0
>         Environment: Hadoop Keberos (Secure) cluster with LinuxContainerExcutor is enabled
> With SPNEGO on for Yarn new RM web services for application submission
> So during application submission xml/json structure was pass webhdfs token
>            Reporter: Karam Singh
>            Assignee: Varun Vasudev
>             Fix For: 2.6.0
>
>
> Encountered this issue during using new YARN's RM WS for application submission, on single
node cluster while submitting Distributed Shell application using RM WS(webservice).
> For this we need  pass custom script and AppMaster jar along with webhdfs token.
> Application was failing with ResouceManager was failing to renew token for user (appOwner).
So RM was Rejecting application with following exception trace in RM log:
> {code}
> 2014-08-19 03:12:54,733 WARN  security.DelegationTokenRenewer (DelegationTokenRenewer.java:handleDTRenewerAppSubmitEvent(661))
- Unable to add the application to the delegation token renewer.
> java.io.IOException: Failed to renew token: Kind: WEBHDFS delegation, Service: <NNHOST>:<FSPORT>,
Ident: (WEBHDFS delegation token 2222 for hrt_qa)
>         at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:394)
>         at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$5(DelegationTokenRenewer.java:357)
>         at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:657)
>         at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:638)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Unexpected HTTP response: code=-1 != 200, op=RENEWDELEGATIONTOKEN,
message=null
>         at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:331)
>         at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:90)
>         at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:598)
>         at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:448)
>         at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:477)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
>         at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:473)
>         at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.renewDelegationToken(WebHdfsFileSystem.java:1318)
>         at org.apache.hadoop.hdfs.web.TokenAspect$TokenManager.renew(TokenAspect.java:73)
>         at org.apache.hadoop.security.token.Token.renew(Token.java:377)
>         at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:477)
>         at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:1)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
>         at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.renewToken(DelegationTokenRenewer.java:473)
>         at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:392)
>         ... 6 more
> Caused by: java.io.IOException: The error stream is null.
>         at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.jsonParse(WebHdfsFileSystem.java:304)
>         at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:329)
>         ... 24 more
> 2014-08-19 03:12:54,735 DEBUG event.AsyncDispatcher (AsyncDispatcher.java:dispatch(164))
- Dispatching the event org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppRejectedEvent.EventType:
APP_REJECTED
> {code}
> From exception trace it is clear that RM is try contact to Namenode on FSPort instead
of Http port and failing to renew token
>  Looks like it is because WebHDFS token Namenodes IP and FSPort in delegation token instead
of http. Causing RM to contact WebHDFS on FSPort and failing to renew token



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message