flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Austin Cawley-Edwards <austin.caw...@gmail.com>
Subject Re: okio and okhttp not shaded in the Flink Uber Jar on EMR
Date Thu, 28 Feb 2019 14:54:08 GMT
Hi Gary,

No, I am running a YARN session (which I start with: flink-yarn-session
--slots 4 --taskManagerMemory 16GB --jobManagerMemory 3GB --detached) and
submit jobs through the REST interface. Thank you for the tips - I will
probably shade it on my side. Is there an official location that the uber
jar dependencies are documented that I can reference for future dependency
additions?

Best,
Austin

On Thu, Feb 28, 2019 at 7:13 AM Gary Yao <gary@ververica.com> wrote:

> Hi Austin,
>
> Are you running your job detached in a per-job cluster? In that case
> inverted
> class loading does not work. This is because we add the user jar to the
> system
> class path, and there is no dynamic class loading involved at the moment
> [1].
>
> You can try the YARN session mode, or – as Chesnay already suggested –
> shade
> the dependency on your side.
>
> Best,
> Gary
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/debugging_classloading.html#overview-of-classloading-in-flink
>
>
> On Wed, Feb 27, 2019 at 8:57 PM Austin Cawley-Edwards <
> austin.cawley@gmail.com> wrote:
>
>> Thanks Gary,
>>
>> I will try to look into why the child-first strategy seems to have failed
>> for this dependency.
>>
>> Best,
>> Austin
>>
>> On Wed, Feb 27, 2019 at 12:25 PM Gary Yao <gary@ververica.com> wrote:
>>
>>> Hi,
>>>
>>> Actually Flink's inverted class loading feature was designed to mitigate
>>> problems with different versions of libraries that are not compatible
>>> with
>>> each other [1]. You may want to debug why it does not work for you.
>>>
>>> You can also try to use the Hadoop free Flink distribution, and export
>>> the
>>> HADOOP_CLASSPATH variable [2].
>>>
>>> Best,
>>> Gary
>>>
>>> [1]
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/debugging_classloading.html#inverted-class-loading-and-classloader-resolution-order
>>> [2]
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/deployment/hadoop.html#configuring-flink-with-hadoop-classpaths
>>>
>>> On Wed, Feb 27, 2019 at 5:23 AM Austin Cawley-Edwards <
>>> austin.cawley@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I recently experienced versioning clashes with the okio and okhttp when
>>>> trying to deploy a Flink 1.6.0 app to AWS EMR on Hadoop 2.8.4. After
>>>> investigating and talking to the okio team (see this issue)
>>>> <https://github.com/square/okio/issues/559>, I found that both okio
>>>> and okhttp exist in the Flink uber jar with versions 1.4.0 and 2.4.0,
>>>> respectively, whereas I'm including versions 2.2.2 and 3.13.1 in my shaded
>>>> jar. The okio team suggested that Flink should shade the uber jar to fix
>>>> the issue, but I'm wondering if there is something I can do on my end to
>>>> have all versions exist simultaneously.
>>>>
>>>> From the issue, here are the okio contents of the uber jar:
>>>>
>>>> *jar -tf flink-shaded-hadoop2-uber-1.6.0.jar | grep okio*
>>>>
>>>> META-INF/maven/com.squareup.okio/
>>>> META-INF/maven/com.squareup.okio/okio/
>>>> META-INF/maven/com.squareup.okio/okio/pom.properties
>>>> META-INF/maven/com.squareup.okio/okio/pom.xml
>>>> okio/
>>>> okio/AsyncTimeout$1.class
>>>> okio/AsyncTimeout$2.class
>>>> okio/AsyncTimeout$Watchdog.class
>>>> okio/AsyncTimeout.class
>>>> okio/Base64.class
>>>> okio/Buffer$1.class
>>>> okio/Buffer$2.class
>>>> okio/Buffer.class
>>>> okio/BufferedSink.class
>>>> okio/BufferedSource.class
>>>> okio/ByteString.class
>>>> okio/DeflaterSink.class
>>>> okio/ForwardingSink.class
>>>> okio/ForwardingSource.class
>>>> okio/ForwardingTimeout.class
>>>> okio/GzipSink.class
>>>> okio/GzipSource.class
>>>> okio/InflaterSource.class
>>>> okio/Okio$1.class
>>>> okio/Okio$2.class
>>>> okio/Okio$3.class
>>>> okio/Okio.class
>>>> okio/RealBufferedSink$1.class
>>>> okio/RealBufferedSink.class
>>>> okio/RealBufferedSource$1.class
>>>> okio/RealBufferedSource.class
>>>> okio/Segment.class
>>>> okio/SegmentPool.class
>>>> okio/SegmentedByteString.class
>>>> okio/Sink.class
>>>> okio/Source.class
>>>> okio/Timeout$1.class
>>>> okio/Timeout.class
>>>> okio/Util.class
>>>>
>>>> Thank you,
>>>> Austin Cawley-Edwards
>>>>
>>>

Mime
View raw message