flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Pompermaier <pomperma...@okkam.it>
Subject Re: Guava version conflict
Date Thu, 15 Jun 2017 18:24:23 GMT
Hi Gordon,
any news on this?

On Mon, Jun 12, 2017 at 9:54 AM, Tzu-Li (Gordon) Tai <tzulitai@apache.org>
wrote:

> This seems like a shading problem then.
> I’ve tested this again with Maven 3.0.5, even without building against CDH
> Hadoop binaries the flink-dist jar contains non-shaded Guava dependencies.
>
> Let me investigate a bit and get back to this!
>
> Cheers,
> Gordon
>
>
> On 8 June 2017 at 2:47:02 PM, Flavio Pompermaier (pompermaier@okkam.it)
> wrote:
>
> On an empty machine (with Ubuntu 14.04.5 LTS) and an empty maven local
> repo I did:
>
>    1. git clone https://github.com/apache/flink.git && cd flink && git
>    checkout tags/release-1.2.1
>    2. /opt/devel/apache-*maven-3.3.9*/bin/mvn clean install
>    -Dhadoop.version=2.6.0-cdh5.9.0 -Dhbase.version=1.2.0-cdh5.9.0
>    -Dhadoop.core.version=2.6.0-mr1-cdh5.9.0 -DskipTests -Pvendor-repos
>    3. cd flink-dist
>    4. /opt/devel/apache-maven-3.3.9/bin/mvn clean install
>    -Dhadoop.version=2.6.0-cdh5.9.0 -Dhbase.version=1.2.0-cdh5.9.0
>    -Dhadoop.core.version=2.6.0-mr1-cdh5.9.0 -DskipTests -Pvendor-repos
>    5. jar tf target/flink-1.2.1-bin/flink-1.2.1/lib/flink-dist_2.10-1.2.1.jar
>     | grep MoreExecutors
>
> And I still see guava dependencies:
>
> org/apache/flink/hadoop/shaded/com/google/common/util/concur
> rent/MoreExecutors$1.class
> org/apache/flink/hadoop/shaded/com/google/common/util/concur
> rent/MoreExecutors$SameThreadExecutorService.class
> org/apache/flink/hadoop/shaded/com/google/common/util/concur
> rent/MoreExecutors$ListeningDecorator.class
> org/apache/flink/hadoop/shaded/com/google/common/util/concur
> rent/MoreExecutors$ScheduledListeningDecorator.class
> org/apache/flink/hadoop/shaded/com/google/common/util/concur
> rent/MoreExecutors.class
> org/apache/flink/shaded/com/google/common/util/concurrent/Mo
> reExecutors$1.class
> org/apache/flink/shaded/com/google/common/util/concurrent/Mo
> reExecutors$2.class
> org/apache/flink/shaded/com/google/common/util/concurrent/Mo
> reExecutors$3.class
> org/apache/flink/shaded/com/google/common/util/concurrent/Mo
> reExecutors$4.class
> org/apache/flink/shaded/com/google/common/util/concurrent/Mo
> reExecutors$Application$1.class
> org/apache/flink/shaded/com/google/common/util/concurrent/Mo
> reExecutors$Application.class
> org/apache/flink/shaded/com/google/common/util/concurrent/Mo
> reExecutors$DirectExecutor.class
> org/apache/flink/shaded/com/google/common/util/concurrent/Mo
> reExecutors$DirectExecutorService.class
> org/apache/flink/shaded/com/google/common/util/concurrent/Mo
> reExecutors$ListeningDecorator.class
> org/apache/flink/shaded/com/google/common/util/concurrent/Mo
> reExecutors$ScheduledListeningDecorator$ListenableScheduledTask.class
> org/apache/flink/shaded/com/google/common/util/concurrent/Mo
> reExecutors$ScheduledListeningDecorator$NeverSuccessfulListe
> nableFutureTask.class
> org/apache/flink/shaded/com/google/common/util/concurrent/Mo
> reExecutors$ScheduledListeningDecorator.class
> org/apache/flink/shaded/com/google/common/util/concurrent/Mo
> reExecutors.class
> com/google/common/util/concurrent/MoreExecutors$1.class
> com/google/common/util/concurrent/MoreExecutors$SameThreadEx
> ecutorService.class
> com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class
> com/google/common/util/concurrent/MoreExecutors$ScheduledLis
> teningDecorator.class
> com/google/common/util/concurrent/MoreExecutors.class
>
> It seems that it mix togheter guava 11 (probably coming from CDH
> dependencies) and guava 18 classes.
>
> Also using *maven 3.0.5* lead to the same output :(
>
> Best,
> Flavio
>
> On Wed, Jun 7, 2017 at 6:21 PM, Tzu-Li (Gordon) Tai <tzulitai@apache.org>
> wrote:
>
>> Yes, those should not be in the flink-dist jar, so the root reason should
>> be that the shading isn’t working properly for your custom build.
>>
>> If possible, could you try building Flink again with a lower Maven
>> version as specified in the doc, and see if that works?
>> If so, it could be that Maven 3.3.x simply isn’t shading properly even
>> with the double compilation trick.
>>
>>
>> On 7 June 2017 at 6:17:15 PM, Flavio Pompermaier (pompermaier@okkam.it)
>> wrote:
>>
>> What I did was to take the sources of the new ES connector and I took
>> them into my code.
>> Flink was compiled with maven 3.3+ but I did the double compilation as
>> specified in the Flink build section.
>> In flink dist I see guava classes, e.g.:
>>
>> com/google/common/util/concurrent/MoreExecutors$1.class
>> com/google/common/util/concurrent/MoreExecutors$SameThreadEx
>> ecutorService.class
>> com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class
>> com/google/common/util/concurrent/MoreExecutors$ScheduledLis
>> teningDecorator.class
>> com/google/common/util/concurrent/MoreExecutors.class
>>
>> Is it a problem of the shading with Maven 3.3+?
>>
>> Best,
>> Flavio
>>
>> On Wed, Jun 7, 2017 at 5:48 PM, Tzu-Li (Gordon) Tai <tzulitai@apache.org>
>> wrote:
>>
>>> Ah, I assumed you were running 1.3.0 (since you mentioned “new” ES
>>> connector).
>>>
>>> Another thing to check, if you built Flink yourself, make sure you’re
>>> not using Maven 3.3+. There are shading problems when Flink is built with
>>> Maven versions higher then that.
>>> The flink-dist jar should not contain any non-shaded Guava dependencies,
>>> could you also quickly check that?
>>>
>>> On 7 June 2017 at 5:42:28 PM, Flavio Pompermaier (pompermaier@okkam.it)
>>> wrote:
>>>
>>> I shaded the Elasticsearch dependency [1] and now the job works.
>>> So I cannot run a job that needs guava 18 on Flink 1.2.1...
>>>
>>> [1]  https://www.elastic.co/blog/to-shade-or-not-to-shade
>>>
>>> On Wed, Jun 7, 2017 at 5:33 PM, Tzu-Li (Gordon) Tai <tzulitai@apache.org
>>> > wrote:
>>>
>>>> Hi Flavio,
>>>>
>>>> Could there be another dependency in your job that requires a
>>>> conflicting version (w.r.t. ES 2.4.1) of Guava?
>>>> I’ve just double checked the flink-dist jar, there doesn’t seem to be
>>>> any non-shaded Guava dependencies there, so the conflict should not have
>>>> been caused by Flink.
>>>>
>>>> Cheers,
>>>> Gordon
>>>>
>>>>
>>>> On 7 June 2017 at 4:12:04 PM, Flavio Pompermaier (pompermaier@okkam.it)
>>>> wrote:
>>>>
>>>> Hi to all,
>>>> I'm trying to use the new ES connector to index data from Flink (with
>>>> ES 2.4.1).
>>>> When I try to run it from Eclipse everything is ok, when I run it from
>>>> the cluster I get the following exception:
>>>>
>>>> java.lang.NoSuchMethodError: com.google.common.util.concurr
>>>> ent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
>>>>         at org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.
>>>> java:192)
>>>>         at org.elasticsearch.client.transport.TransportClient$Builder.b
>>>> uild(TransportClient.java:131)
>>>>
>>>> In my fat jar there are the classes of guava 18 (ES requires that
>>>> version), Flink runs on CDH 5.9 (that use guava 11), in flink-dist jar I
>>>> think that there's guava 11 classes while in flink-hadoop-compatibility
>>>> there are shade guava 18 dependencies.
>>>>
>>>> How can I make the job successfully run on the cluster?
>>>>
>>>> Best,
>>>> Flavio
>>>>
>>>>
>>>
>

Mime
View raw message