hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jian He (JIRA)" <j...@apache.org>
Subject [jira] [Reopened] (YARN-2425) When Application submitted by via Yarn RM WS, log aggregation does not happens
Date Tue, 09 Dec 2014 18:54:13 GMT

     [ https://issues.apache.org/jira/browse/YARN-2425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jian He reopened YARN-2425:
---------------------------

turns out YARN-2704 doesn't solve this problem completely. 
there are race conditions that RM could try to launch the AM container or AM tries to launch
the task container before NM actually receives the token from RM. reopen this.

> When Application submitted by via Yarn RM WS, log aggregation does not happens
> ------------------------------------------------------------------------------
>
>                 Key: YARN-2425
>                 URL: https://issues.apache.org/jira/browse/YARN-2425
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: log-aggregation
>    Affects Versions: 2.5.0, 2.6.0
>         Environment: Secure (Kerberos enabled) hadoop cluster. With SPNEGO for Yarn RM
enabled
>            Reporter: Karam Singh
>            Assignee: Varun Vasudev
>
> When submit App to Yarn RM using Web service we need to pass credentials/tokens in json
object/xml object to webservice
> As HDFS namenode does not provides any DT over WS (base64 encoded) like webhdfs/timeline
server does. (HDFS fetch dt commad fetch java writable object and writes it to target file,
we we cannot forward via application Submission WS objects)
> Looks like there is not way to pass HDFS token to NodeManager. 
> While starting Application container also tries to create Application log aggregation
dir and fails with following type exception
> {code}
> java.io.IOException: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException:
Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "<hostname>/<ip>";
destination host is: "<NameNodeHost>":<FSPort>;
>         at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1415)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1364)
>         at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>         at com.sun.proxy.$Proxy34.getFileInfo(Unknown Source)
>         at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:725)
>         at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
>         at com.sun.proxy.$Proxy35.getFileInfo(Unknown Source)
>         at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1781)
>         at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1069)
>         at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1065)
>         at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>         at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1065)
>         at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.checkExists(LogAggregationService.java:240)
>         at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.access$100(LogAggregationService.java:64)
>         at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService$1.run(LogAggregationService.java:268)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
>         at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.createAppDir(LogAggregationService.java:253)
>         at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.initAppAggregator(LogAggregationService.java:344)
>         at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.initApp(LogAggregationService.java:310)
>         at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:421)
>         at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:64)
>         at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
>         at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client
cannot authenticate via:[TOKEN, KERBEROS]
>         at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:679)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
>         at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:642)
>         at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:725)
>         at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367)
>         at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1382)
>         ... 29 more
> Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate
via:[TOKEN, KERBEROS]
>         at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:172)
>         at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:396)
>         at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:552)
>         at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:367)
>         at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:717)
>         at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:713)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
>         at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:712)
>         ... 32 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message