flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kidong Lee <mykid...@gmail.com>
Subject Re: Jar hell when running streaming job in YARN session
Date Thu, 15 Dec 2016 10:18:32 GMT
To avoid guava conflict, I use maven shade plugin to package my fat jar.
If you use maven, the shade plugin looks like this:


To package fat jar:

mvn -e -DskipTests=true clean install shade:shade;

I hope, it helps.

- Kidong Lee.

2016-12-15 19:04 GMT+09:00 Yury Ruchin <yuri.ruchin@gmail.com>:

> Hi,
> I have run into a classpath issue when running Flink streaming job in YARN
> session. I package my app into a fat jar with all the dependencies needed.
> One of them is Google Guava. I then submit the jar to the session. The task
> managers pre-created by the session build their classpath from the
> FLINK_LIB_DIR and Hadoop / YARN lib directories. Unfortunately, there is a
> dated Guava version pulled along with Hadoop dependencies which conflicts
> with the version my app needs. Even worse, the Flink lib dir and Hadoop
> libraries take precedence over my jar.
> If I remember correctly, in Spark there is an option meaning "user
> classpath goes first when looking for classes". I couldn't find anything
> similar for Flink. I tried "flink run -C file:///path/to/my/libraries" in
> the hope to extend the classpath but the old Guava version takes precedence
> anyway.
> How else can I bump "priority" of my app jar during the load process?
> Thanks,
> Yury

View raw message