flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai" <tzuli...@apache.org>
Subject Re: Guava version conflict
Date Wed, 07 Jun 2017 16:21:48 GMT
Yes, those should not be in the flink-dist jar, so the root reason should be that the shading
isn’t working properly for your custom build.

If possible, could you try building Flink again with a lower Maven version as specified in
the doc, and see if that works?
If so, it could be that Maven 3.3.x simply isn’t shading properly even with the double compilation
trick.


On 7 June 2017 at 6:17:15 PM, Flavio Pompermaier (pompermaier@okkam.it) wrote:

What I did was to take the sources of the new ES connector and I took them into my code.
Flink was compiled with maven 3.3+ but I did the double compilation as specified in the Flink
build section.
In flink dist I see guava classes, e.g.:

com/google/common/util/concurrent/MoreExecutors$1.class
com/google/common/util/concurrent/MoreExecutors$SameThreadExecutorService.class
com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class
com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class
com/google/common/util/concurrent/MoreExecutors.class

Is it a problem of the shading with Maven 3.3+?

Best,
Flavio

On Wed, Jun 7, 2017 at 5:48 PM, Tzu-Li (Gordon) Tai <tzulitai@apache.org> wrote:
Ah, I assumed you were running 1.3.0 (since you mentioned “new” ES connector).

Another thing to check, if you built Flink yourself, make sure you’re not using Maven 3.3+.
There are shading problems when Flink is built with Maven versions higher then that.
The flink-dist jar should not contain any non-shaded Guava dependencies, could you also quickly
check that?

On 7 June 2017 at 5:42:28 PM, Flavio Pompermaier (pompermaier@okkam.it) wrote:

I shaded the Elasticsearch dependency [1] and now the job works.
So I cannot run a job that needs guava 18 on Flink 1.2.1...

[1]  https://www.elastic.co/blog/to-shade-or-not-to-shade

On Wed, Jun 7, 2017 at 5:33 PM, Tzu-Li (Gordon) Tai <tzulitai@apache.org> wrote:
Hi Flavio,

Could there be another dependency in your job that requires a conflicting version (w.r.t.
ES 2.4.1) of Guava?
I’ve just double checked the flink-dist jar, there doesn’t seem to be any non-shaded Guava
dependencies there, so the conflict should not have been caused by Flink.

Cheers,
Gordon


On 7 June 2017 at 4:12:04 PM, Flavio Pompermaier (pompermaier@okkam.it) wrote:

Hi to all,
I'm trying to use the new ES connector to index data from Flink (with ES 2.4.1).
When I try to run it from Eclipse everything is ok, when I run it from the cluster I get the
following exception:

java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
        at org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.java:192)
        at org.elasticsearch.client.transport.TransportClient$Builder.build(TransportClient.java:131)

In my fat jar there are the classes of guava 18 (ES requires that version), Flink runs on
CDH 5.9 (that use guava 11), in flink-dist jar I think that there's guava 11 classes while
in flink-hadoop-compatibility there are shade guava 18 dependencies.

How can I make the job successfully run on the cluster?

Best,
Flavio



Mime
View raw message