flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: flink build issue
Date Tue, 27 Jan 2015 10:07:18 GMT
Hi,

thank you for trying out Flink.
I'm sorry that you ran into this issue. Flink does not have support for
Hadoop YARN before 2.2.0.
The reason for that is that Hadoop has changed the YARN APIs with the 2.2.0
release. (the pre 2.2.0 APIs are marked as "alpha")
If you take a closer look into the Spark code, you'll see that they have
two implementations of their YARN client:
https://github.com/apache/spark/tree/branch-1.2/yarn ("alpha" and "stable"
directories).

As you can see in this JIRA, Spark is dropping support for the pre-2.2.0
YARN APIs with their next release (1.3.0):
https://issues.apache.org/jira/browse/SPARK-3445

So much for the bad news.
The good news is, that you can still use Flink on CDH 4.3.0, if you install
it directly in your cluster.
This basically means that you have to copy the flink directory to all
machines of your cluster (I usually use rsync for that).
For using Flink as a stand alone installation on CDH 4.3.0, you have to
build it like this:

     mvn clean install -P\!include-yarn -Pvendor-repos
-Dhadoop.version=2.0.0-cdh4.3.0  -DskipTests

This is necessary because the HDFS client of Flink needs to match the
version of HDFS in CDH 4.3.

Please let me know if you need further help getting Flink to run in your
cluster!


Best,
Robert





On Tue, Jan 27, 2015 at 10:24 AM, takun <yongjun.tian@tendcloud.com> wrote:

> Hello :
>
> I want build flink 0.8 on yarn ,but we use version : 2.0.0-cdh4.3.0 ,it
> failed
> how can i build it .
>
> command : mvn clean install -DskipTests -Pvendor-repos
> -Dhadoop.version=2.0.0-cdh4.3.0
>
> The document say , not support before hadoop 2.2.0 ,but i want to run on
> yarn like spark on yarn .
> Could someone help me ,or some suggestion.
>
> thanks
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.1:compile
> (default-compile) on project flink-yarn: Compilation failure: Compilation
> failure:
> [ERROR]
> /Users/tianyongjun/work/source/flink/flink-0.8.0/flink-addons/flink-yarn/src/main/java/org/apache/flink/yarn/appMaster/ApplicationMaster.java:[69,41]
> package org.apache.hadoop.yarn.client.api does not exist
> [ERROR]
> /Users/tianyongjun/work/source/flink/flink-0.8.0/flink-addons/flink-yarn/src/main/java/org/apache/flink/yarn/appMaster/ApplicationMaster.java:[70,52]
> package org.apache.hadoop.yarn.client.api.AMRMClient does not exist
> [ERROR]
> /Users/tianyongjun/work/source/flink/flink-0.8.0/flink-addons/flink-yarn/src/main/java/org/apache/flink/yarn/appMaster/ApplicationMaster.java:[71,41]
> package org.apache.hadoop.yarn.client.api does not exist
> [ERROR]
> /Users/tianyongjun/work/source/flink/flink-0.8.0/flink-addons/flink-yarn/src/main/java/org/apache/flink/yarn/appMaster/ApplicationMaster.java:[120,17]
> cannot find symbol
> [ERROR] symbol:   class AMRMClient
> [ERROR] location: class org.apache.flink.yarn.appMaster.ApplicationMaster
> [ERROR]
> /Users/tianyongjun/work/source/flink/flink-0.8.0/flink-addons/flink-yarn/src/main/java/org/apache/flink/yarn/appMaster/ApplicationMaster.java:[120,28]
> cannot find symbol
> [ERROR] symbol:   class ContainerRequest
> [ERROR] location: class org.apache.flink.yarn.appMaster.ApplicationMaster
> [ERROR]
> /Users/tianyongjun/work/source/flink/flink-0.8.0/flink-addons/flink-yarn/src/main/java/org/apache/flink/yarn/appMaster/ApplicationMaster.java:[125,17]
> cannot find symbol
> [ERROR] symbol:   class NMClient

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message