flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Pompermaier <pomperma...@okkam.it>
Subject Re: Guava version conflict
Date Fri, 16 Jun 2017 10:23:32 GMT
I've never tested with flink 1.3.0, I have the problem with Flink 1.2.1.
Didn't you say that you also had non-shaded guava dependencies in the flink
dist jar some days ago?

On Fri, Jun 16, 2017 at 12:19 PM, Tzu-Li (Gordon) Tai <tzulitai@apache.org>
wrote:

> Hi Flavio,
>
> I was just doing some end-to-end rebuild Flink + cluster execution with ES
> sink tests, and it seems like the Guava shading problem isn’t there anymore
> in the flink-dist jar.
>
> On the `release-1.3` branch, built with Maven 3.0.5, the Guava
> dependencies in flink-dist are all properly shaded.
> Tested with ES 2.3.5 (the default Elasticsearch 2 connector version) and
> ES 2.4.1 (overwritten the ES 2 version and rebuilt the ES connector), all
> worked fine without the Guava conflict issue.
>
> So I’m pretty sure that if the problem still exists for you, the conflict
> would have came from other dependencies in your code.
>
> Cheers,
> Gordon
>
>
> On 15 June 2017 at 8:24:48 PM, Flavio Pompermaier (pompermaier@okkam.it)
> wrote:
>
> Hi Gordon,
> any news on this?
>
> On Mon, Jun 12, 2017 at 9:54 AM, Tzu-Li (Gordon) Tai <tzulitai@apache.org>
> wrote:
>
>> This seems like a shading problem then.
>> I’ve tested this again with Maven 3.0.5, even without building against
>> CDH Hadoop binaries the flink-dist jar contains non-shaded Guava
>> dependencies.
>>
>> Let me investigate a bit and get back to this!
>>
>> Cheers,
>> Gordon
>>
>>
>> On 8 June 2017 at 2:47:02 PM, Flavio Pompermaier (pompermaier@okkam.it)
>> wrote:
>>
>> On an empty machine (with Ubuntu 14.04.5 LTS) and an empty maven local
>> repo I did:
>>
>>    1. git clone https://github.com/apache/flink.git && cd flink &&
git
>>    checkout tags/release-1.2.1
>>    2. /opt/devel/apache-*maven-3.3.9*/bin/mvn clean install
>>    -Dhadoop.version=2.6.0-cdh5.9.0 -Dhbase.version=1.2.0-cdh5.9.0
>>    -Dhadoop.core.version=2.6.0-mr1-cdh5.9.0 -DskipTests -Pvendor-repos
>>    3. cd flink-dist
>>    4. /opt/devel/apache-maven-3.3.9/bin/mvn clean install
>>    -Dhadoop.version=2.6.0-cdh5.9.0 -Dhbase.version=1.2.0-cdh5.9.0
>>    -Dhadoop.core.version=2.6.0-mr1-cdh5.9.0 -DskipTests -Pvendor-repos
>>    5. jar tf target/flink-1.2.1-bin/flink-1.2.1/lib/flink-dist_2.10-1.2.1.jar
>>     | grep MoreExecutors
>>
>> And I still see guava dependencies:
>>
>> org/apache/flink/hadoop/shaded/com/google/common/util/concur
>> rent/MoreExecutors$1.class
>> org/apache/flink/hadoop/shaded/com/google/common/util/concur
>> rent/MoreExecutors$SameThreadExecutorService.class
>> org/apache/flink/hadoop/shaded/com/google/common/util/concur
>> rent/MoreExecutors$ListeningDecorator.class
>> org/apache/flink/hadoop/shaded/com/google/common/util/concur
>> rent/MoreExecutors$ScheduledListeningDecorator.class
>> org/apache/flink/hadoop/shaded/com/google/common/util/concur
>> rent/MoreExecutors.class
>> org/apache/flink/shaded/com/google/common/util/concurrent/Mo
>> reExecutors$1.class
>> org/apache/flink/shaded/com/google/common/util/concurrent/Mo
>> reExecutors$2.class
>> org/apache/flink/shaded/com/google/common/util/concurrent/Mo
>> reExecutors$3.class
>> org/apache/flink/shaded/com/google/common/util/concurrent/Mo
>> reExecutors$4.class
>> org/apache/flink/shaded/com/google/common/util/concurrent/Mo
>> reExecutors$Application$1.class
>> org/apache/flink/shaded/com/google/common/util/concurrent/Mo
>> reExecutors$Application.class
>> org/apache/flink/shaded/com/google/common/util/concurrent/Mo
>> reExecutors$DirectExecutor.class
>> org/apache/flink/shaded/com/google/common/util/concurrent/Mo
>> reExecutors$DirectExecutorService.class
>> org/apache/flink/shaded/com/google/common/util/concurrent/Mo
>> reExecutors$ListeningDecorator.class
>> org/apache/flink/shaded/com/google/common/util/concurrent/Mo
>> reExecutors$ScheduledListeningDecorator$ListenableScheduledTask.class
>> org/apache/flink/shaded/com/google/common/util/concurrent/Mo
>> reExecutors$ScheduledListeningDecorator$NeverSuccessfulListe
>> nableFutureTask.class
>> org/apache/flink/shaded/com/google/common/util/concurrent/Mo
>> reExecutors$ScheduledListeningDecorator.class
>> org/apache/flink/shaded/com/google/common/util/concurrent/Mo
>> reExecutors.class
>> com/google/common/util/concurrent/MoreExecutors$1.class
>> com/google/common/util/concurrent/MoreExecutors$SameThreadEx
>> ecutorService.class
>> com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class
>> com/google/common/util/concurrent/MoreExecutors$ScheduledLis
>> teningDecorator.class
>> com/google/common/util/concurrent/MoreExecutors.class
>>
>> It seems that it mix togheter guava 11 (probably coming from CDH
>> dependencies) and guava 18 classes.
>>
>> Also using *maven 3.0.5* lead to the same output :(
>>
>> Best,
>> Flavio
>>
>> On Wed, Jun 7, 2017 at 6:21 PM, Tzu-Li (Gordon) Tai <tzulitai@apache.org>
>> wrote:
>>
>>> Yes, those should not be in the flink-dist jar, so the root reason
>>> should be that the shading isn’t working properly for your custom build.
>>>
>>> If possible, could you try building Flink again with a lower Maven
>>> version as specified in the doc, and see if that works?
>>> If so, it could be that Maven 3.3.x simply isn’t shading properly even
>>> with the double compilation trick.
>>>
>>>
>>> On 7 June 2017 at 6:17:15 PM, Flavio Pompermaier (pompermaier@okkam.it)
>>> wrote:
>>>
>>> What I did was to take the sources of the new ES connector and I took
>>> them into my code.
>>> Flink was compiled with maven 3.3+ but I did the double compilation as
>>> specified in the Flink build section.
>>> In flink dist I see guava classes, e.g.:
>>>
>>> com/google/common/util/concurrent/MoreExecutors$1.class
>>> com/google/common/util/concurrent/MoreExecutors$SameThreadEx
>>> ecutorService.class
>>> com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class
>>> com/google/common/util/concurrent/MoreExecutors$ScheduledLis
>>> teningDecorator.class
>>> com/google/common/util/concurrent/MoreExecutors.class
>>>
>>> Is it a problem of the shading with Maven 3.3+?
>>>
>>> Best,
>>> Flavio
>>>
>>> On Wed, Jun 7, 2017 at 5:48 PM, Tzu-Li (Gordon) Tai <tzulitai@apache.org
>>> > wrote:
>>>
>>>> Ah, I assumed you were running 1.3.0 (since you mentioned “new” ES
>>>> connector).
>>>>
>>>> Another thing to check, if you built Flink yourself, make sure you’re
>>>> not using Maven 3.3+. There are shading problems when Flink is built with
>>>> Maven versions higher then that.
>>>> The flink-dist jar should not contain any non-shaded Guava
>>>> dependencies, could you also quickly check that?
>>>>
>>>> On 7 June 2017 at 5:42:28 PM, Flavio Pompermaier (pompermaier@okkam.it)
>>>> wrote:
>>>>
>>>> I shaded the Elasticsearch dependency [1] and now the job works.
>>>> So I cannot run a job that needs guava 18 on Flink 1.2.1...
>>>>
>>>> [1]  https://www.elastic.co/blog/to-shade-or-not-to-shade
>>>>
>>>> On Wed, Jun 7, 2017 at 5:33 PM, Tzu-Li (Gordon) Tai <
>>>> tzulitai@apache.org> wrote:
>>>>
>>>>> Hi Flavio,
>>>>>
>>>>> Could there be another dependency in your job that requires a
>>>>> conflicting version (w.r.t. ES 2.4.1) of Guava?
>>>>> I’ve just double checked the flink-dist jar, there doesn’t seem to
be
>>>>> any non-shaded Guava dependencies there, so the conflict should not have
>>>>> been caused by Flink.
>>>>>
>>>>> Cheers,
>>>>> Gordon
>>>>>
>>>>>
>>>>> On 7 June 2017 at 4:12:04 PM, Flavio Pompermaier (pompermaier@okkam.it)
>>>>> wrote:
>>>>>
>>>>> Hi to all,
>>>>> I'm trying to use the new ES connector to index data from Flink (with
>>>>> ES 2.4.1).
>>>>> When I try to run it from Eclipse everything is ok, when I run it from
>>>>> the cluster I get the following exception:
>>>>>
>>>>> java.lang.NoSuchMethodError: com.google.common.util.concurr
>>>>> ent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
>>>>>         at org.elasticsearch.threadpool.ThreadPool.<clinit>(
>>>>> ThreadPool.java:192)
>>>>>         at org.elasticsearch.client.transport.TransportClient$
>>>>> Builder.build(TransportClient.java:131)
>>>>>
>>>>> In my fat jar there are the classes of guava 18 (ES requires that
>>>>> version), Flink runs on CDH 5.9 (that use guava 11), in flink-dist jar
I
>>>>> think that there's guava 11 classes while in flink-hadoop-compatibility
>>>>> there are shade guava 18 dependencies.
>>>>>
>>>>> How can I make the job successfully run on the cluster?
>>>>>
>>>>> Best,
>>>>> Flavio
>>>>>
>>>>>
>>>>
>>
>

Mime
View raw message