flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Pompermaier <pomperma...@okkam.it>
Subject Re: Guava version conflict
Date Wed, 07 Jun 2017 16:16:42 GMT
What I did was to take the sources of the new ES connector and I took them
into my code.
Flink was compiled with maven 3.3+ but I did the double compilation as
specified in the Flink build section.
In flink dist I see guava classes, e.g.:

com/google/common/util/concurrent/MoreExecutors$1.class
com/google/common/util/concurrent/MoreExecutors$SameThreadExecutorService.class
com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class
com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class
com/google/common/util/concurrent/MoreExecutors.class

Is it a problem of the shading with Maven 3.3+?

Best,
Flavio

On Wed, Jun 7, 2017 at 5:48 PM, Tzu-Li (Gordon) Tai <tzulitai@apache.org>
wrote:

> Ah, I assumed you were running 1.3.0 (since you mentioned “new” ES
> connector).
>
> Another thing to check, if you built Flink yourself, make sure you’re not
> using Maven 3.3+. There are shading problems when Flink is built with Maven
> versions higher then that.
> The flink-dist jar should not contain any non-shaded Guava dependencies,
> could you also quickly check that?
>
> On 7 June 2017 at 5:42:28 PM, Flavio Pompermaier (pompermaier@okkam.it)
> wrote:
>
> I shaded the Elasticsearch dependency [1] and now the job works.
> So I cannot run a job that needs guava 18 on Flink 1.2.1...
>
> [1]  https://www.elastic.co/blog/to-shade-or-not-to-shade
>
> On Wed, Jun 7, 2017 at 5:33 PM, Tzu-Li (Gordon) Tai <tzulitai@apache.org>
> wrote:
>
>> Hi Flavio,
>>
>> Could there be another dependency in your job that requires a conflicting
>> version (w.r.t. ES 2.4.1) of Guava?
>> I’ve just double checked the flink-dist jar, there doesn’t seem to be any
>> non-shaded Guava dependencies there, so the conflict should not have been
>> caused by Flink.
>>
>> Cheers,
>> Gordon
>>
>>
>> On 7 June 2017 at 4:12:04 PM, Flavio Pompermaier (pompermaier@okkam.it)
>> wrote:
>>
>> Hi to all,
>> I'm trying to use the new ES connector to index data from Flink (with ES
>> 2.4.1).
>> When I try to run it from Eclipse everything is ok, when I run it from
>> the cluster I get the following exception:
>>
>> java.lang.NoSuchMethodError: com.google.common.util.concurr
>> ent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
>>         at org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.
>> java:192)
>>         at org.elasticsearch.client.transport.TransportClient$Builder.b
>> uild(TransportClient.java:131)
>>
>> In my fat jar there are the classes of guava 18 (ES requires that
>> version), Flink runs on CDH 5.9 (that use guava 11), in flink-dist jar I
>> think that there's guava 11 classes while in flink-hadoop-compatibility
>> there are shade guava 18 dependencies.
>>
>> How can I make the job successfully run on the cluster?
>>
>> Best,
>> Flavio
>>
>>
>

Mime
View raw message