flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-6376) when deploy flink cluster on the yarn, it is lack of hdfs delegation token.
Date Wed, 21 Jun 2017 05:48:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-6376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16057023#comment-16057023
] 

ASF GitHub Bot commented on FLINK-6376:
---------------------------------------

Github user Rucongzhang commented on the issue:

    https://github.com/apache/flink/pull/3776
  
    @tzulitai ,When I have fixed the problem, I want to write a IT case. But, in the Yarn
IT case:
    1、 It uses the yarn mini cluster, which is for testing.I do not know whether it is using
HDFS Delegation token or not.
    2、And what's more, the HDFS Delegation Token is used by yarn node manager. It is difficult
to judge whether to have this token or not in yarn client. The token is set into yarn application
context,but the yarn client does not have API to get yarn application context.
    How do you think ? Thanks a lot!


> when deploy flink cluster on the yarn, it is lack of hdfs delegation token.
> ---------------------------------------------------------------------------
>
>                 Key: FLINK-6376
>                 URL: https://issues.apache.org/jira/browse/FLINK-6376
>             Project: Flink
>          Issue Type: Bug
>          Components: Security, YARN
>            Reporter: zhangrucong1982
>            Assignee: zhangrucong1982
>
> 1、I use the flink of version 1.2.0. And  I deploy the flink cluster on the yarn. The
hadoop version is 2.7.2.
> 2、I use flink in security model with the keytab and principal. And the key configuration
is :security.kerberos.login.keytab: /home/ketab/test.keytab 、security.kerberos.login.principal:
test.
> 3、The yarn configuration is default and enable the yarn log aggregation configuration"
yarn.log-aggregation-enable : true";
> 4、 Deploying the flink cluster  on the yarn,  the yarn Node manager occur the following
failure when aggregation the log in HDFS. The basic reason is lack of HDFS  delegation token.

>  java.io.IOException: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException:
Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "SZV1000258954/10.162.181.24";
destination host is: "SZV1000258954":25000;
>         at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:796)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1515)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1447)
>         at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
>         at com.sun.proxy.$Proxy26.getFileInfo(Unknown Source)
>         at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:802)
>         at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:201)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
>         at com.sun.proxy.$Proxy27.getFileInfo(Unknown Source)
>         at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1919)
>         at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1500)
>         at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1496)
>         at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>         at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1496)
>         at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.checkExists(LogAggregationService.java:271)
>         at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.access$100(LogAggregationService.java:68)
>         at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService$1.run(LogAggregationService.java:299)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1769)
>         at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.createAppDir(LogAggregationService.java:284)
>         at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.initAppAggregator(LogAggregationService.java:390)
>         at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.initApp(LogAggregationService.java:342)
>         at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:470)
>         at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:68)
>         at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:194)
>         at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:120)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client
cannot authenticate via:[TOKEN, KERBEROS]
>         at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:722)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1769)
>         at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:685)
>         at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:772)
>         at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:394)
>         at org.apache.hadoop.ipc.Client.getConnection(Client.java:1564)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1486)
>         ... 29 more
> Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate
via:[TOKEN, KERBEROS]
>         at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:177)
>         at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:404)
>         at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:581)
>         at org.apache.hadoop.ipc.Client$Connection.access$1900(Client.java:394)
>         at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:764)
>         at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:760)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1769)
>         at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:759)
>         ... 32 more
> 5、the hadoop fix the hadoop issue 14116(https://issues.apache.org/jira/browse/HADOOP-14116),
if there is no HDFS  delegation token, it will try 20 times after sleeping 1 second. So it
will cause the flink cluster deploy on yarn is very slowly, it will spent about 5 minutes
to deploy the cluster with 2 taskmanagers.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message