flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chesnay Schepler <ches...@apache.org>
Subject Re: Flink session on Yarn - ClassNotFoundException
Date Tue, 29 Aug 2017 16:30:52 GMT
Hello,

The ClassNotFoundException indicates that you are using a Flink version 
that wasn't compiled against hadoop 2.7.
Replacing one part of the hadoop dependency will most likely not cut it 
(or fail mysteriously down the line),
so i would suggest to check the downloads 
<http://flink.apache.org/downloads.html> page and pick the binary 
matching your hadoop version.

It should be enough to replace the flink-shaded-hadoop2-uber-1.3.2.jar 
under /lib.

On 29.08.2017 11:46, Albert Giménez wrote:
> Hi,
>
> I’m trying to start a flink (1.3.2) session as explained in the docs 
> (https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/yarn_setup.html#start-a-session),

> but I keep getting the “ClassNotFoundException” you can see below.
>
> I’m running an HDInsight cluster on Azure (HDFS and YARN v2.7.3), and 
> I’m exporting my HADOOP_CONF_DIR before running the yarn-session command.
>
> Error while deploying YARN cluster: Couldn't deploy Yarn cluster
> java.lang.RuntimeException: Couldn't deploy Yarn cluster
> at 
> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(AbstractYarnClusterDescriptor.java:443)
> at 
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:630)
> at 
> org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:486)
> at 
> org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:483)
> at 
> org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at 
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
> at 
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:483)
> Caused by: java.lang.RuntimeException: java.lang.RuntimeException: 
> java.lang.ClassNotFoundException: Class 
> org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider 
> not found
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2227)
> at 
> org.apache.hadoop.yarn.client.RMProxy.createRMFailoverProxyProvider(RMProxy.java:161)
> at org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:94)
> at 
> org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:72)
> at 
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceStart(YarnClientImpl.java:187)
> at 
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at 
> org.apache.flink.yarn.AbstractYarnClusterDescriptor.getYarnClient(AbstractYarnClusterDescriptor.java:381)
> at 
> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:458)
> at 
> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(AbstractYarnClusterDescriptor.java:441)
> ... 9 more
> Caused by: java.lang.RuntimeException: 
> java.lang.ClassNotFoundException: Class 
> org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider 
> not found
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2219)
> ... 17 more
> Caused by: java.lang.ClassNotFoundException: Class 
> org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider 
> not found
> at 
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
> ... 18 more
>
>
> So far, I’ve check that the yarn class path is correct (it is), I also 
> tried manually linking the “hadoop-yarn-common" jar from my /usr/hdp 
> directories into Flink’s “lib” directory. In that case, I get an 
> “IllegalAccessError”:
>
> Exception in thread "main" java.lang.IllegalAccessError: tried to 
> access method 
> org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider.getProxyInternal()Ljava/lang/Object;

> from class 
> org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider
> at 
> org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider.init(RequestHedgingRMFailoverProxyProvider.java:75)
> at 
> org.apache.hadoop.yarn.client.RMProxy.createRMFailoverProxyProvider(RMProxy.java:163)
> at org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:94)
> at 
> org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:72)
> at 
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceStart(YarnClientImpl.java:187)
> at 
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at 
> org.apache.flink.yarn.AbstractYarnClusterDescriptor.getYarnClient(AbstractYarnClusterDescriptor.java:381)
> at 
> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:458)
> at 
> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(AbstractYarnClusterDescriptor.java:441)
> at 
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:630)
> at 
> org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:486)
> at 
> org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:483)
> at 
> org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at 
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
> at 
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:483)
>
> Any ideas on what might be going on? This last error makes me think 
> that maybe the error has to do with the HDFS version Flink is compiled 
> against, but I’m not sure.
>
> Thanks a lot!



Mime
View raw message