flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: Return of Flink shading problems in 1.2.0
Date Mon, 20 Mar 2017 09:34:33 GMT
Here is the JIRA: https://issues.apache.org/jira/browse/FLINK-6125

On Mon, Mar 20, 2017 at 10:27 AM, Robert Metzger <rmetzger@apache.org>
wrote:

> Hi Craig,
>
> I was able to reproduce the issue with maven 3.3 in Flink 1.2. I'll look
> into it.
>
> On Fri, Mar 17, 2017 at 11:56 PM, Foster, Craig <foscraig@amazon.com>
> wrote:
>
>> Ping. So I’ve built with 3.0.5 and it does give proper shading. So it
>> does get me yet another workaround where my only recourse is to use a max
>> version of Maven. Still, I feel there should be a long-term fix at some
>> point in time.
>>
>>
>>
>> I also believe there is a regression in Flink 1.2.0 for Maven 3.3.x with
>> the process as documented, so hoping someone can at least duplicate or let
>> me know of a new workaround for 3.3.x.
>>
>>
>>
>> Thanks!
>>
>> Craig
>>
>>
>>
>> *From: *"Foster, Craig" <foscraig@amazon.com>
>> *Reply-To: *"user@flink.apache.org" <user@flink.apache.org>
>> *Date: *Friday, March 17, 2017 at 7:23 AM
>> *To: *"user@flink.apache.org" <user@flink.apache.org>
>> *Cc: *Ufuk Celebi <uce@apache.org>, Robert Metzger <rmetzger@apache.org>,
>> Stephan Ewen <sewen@apache.org>
>> *Subject: *Re: Return of Flink shading problems in 1.2.0
>>
>>
>>
>> Hey Stephen:
>>
>> I am building twice in every case described in my previous mail. Well,
>> building then rebuilding the flink-dist submodule.
>>
>>
>>
>> This was fixed in BigTop but I started seeing this issue again with Flink
>> 1.2.0. I was wondering if there's something else in the environment that
>> could prevent the shading from working because it isn't now even with the
>> workaround.
>>
>>
>> On Mar 17, 2017, at 4:08 AM, Stephan Ewen <sewen@apache.org> wrote:
>>
>> Hi Craig!
>>
>>
>>
>> Maven 3.3.x has a shading problem. You need to build two times, once from
>> root, once inside "flink-dist". Have a look here:
>>
>>
>>
>> https://ci.apache.org/projects/flink/flink-docs-release-1.2/
>> setup/building.html#dependency-shading
>>
>>
>>
>> Maybe that way missed in BigTop?
>>
>>
>>
>> I am wondering if we should actually throw an error if building with
>> Maven 3.3.x - too many users run into that issue.
>>
>>
>>
>> Stephan
>>
>>
>>
>>
>>
>>
>>
>> On Fri, Mar 17, 2017 at 8:14 AM, Ufuk Celebi <uce@apache.org> wrote:
>>
>> Pulling in Robert and Stephan who know the project's shading setup the
>> best.
>>
>>
>> On Fri, Mar 17, 2017 at 6:52 AM, Foster, Craig <foscraig@amazon.com>
>> wrote:
>> > Hi:
>> >
>> > A few months ago, I was building Flink and ran into shading issues for
>> > flink-dist as described in your docs. We resolved this in BigTop by
>> adding
>> > the correct way to build flink-dist in the do-component-build script and
>> > everything was fine after that.
>> >
>> >
>> >
>> > Now, I’m running into issues doing the same now in Flink 1.2.0 and I’m
>> > trying to figure out what’s changed and how to fix it. Here’s how the
>> > flink-dist jar looks with proper shading:
>> >
>> >
>> >
>> > jar -tvf /usr/lib/flink/lib/flink-dist_2.10-1.1.4.jar | grep
>> > HttpConnectionParams
>> > 2485 Tue Jan 01 00:00:00 UTC 1980
>> > org/apache/flink/hadoop/shaded/org/apache/commons/httpclient
>> /params/HttpConnectionParams.class
>> > 3479 Tue Jan 01 00:00:00 UTC 1980
>> > org/apache/flink/hadoop/shaded/org/apache/http/params/HttpCo
>> nnectionParams.class
>> >
>> >
>> >
>> > When I build Flink 1.2.0 in BigTop, here’s shading for the jar found in
>> the
>> > RPM:
>> >
>> >
>> >
>> > jar -tvf flink-dist_2.10-1.2.0.jar | grep HttpConnectionParams
>> > 2392 Tue Jan 01 00:00:00 GMT 1980
>> > org/apache/commons/httpclient/params/HttpConnectionParams.class
>> > 2485 Tue Jan 01 00:00:00 GMT 1980
>> > org/apache/flink/hadoop/shaded/org/apache/commons/httpclient
>> /params/HttpConnectionParams.class
>> > 3479 Tue Jan 01 00:00:00 GMT 1980
>> > org/apache/flink/hadoop/shaded/org/apache/http/params/HttpCo
>> nnectionParams.class
>> > 2868 Tue Jan 01 00:00:00 GMT 1980
>> > org/apache/http/params/HttpConnectionParams.class
>> >
>> >
>> >
>> > I thought maybe it was some strange thing going on with BigTop, so then
>> I
>> > tried just straight building Flink 1.2.0 (outside BigTop) and get the
>> same
>> > shading:
>> >
>> >
>> >
>> > jar -tvf flink-dist_2.10-1.2.0.jar | grep HttpConnectionParams
>> >
>> >   2485 Fri Mar 17 05:41:16 GMT 2017
>> > org/apache/flink/hadoop/shaded/org/apache/commons/httpclient
>> /params/HttpConnectionParams.class
>> >
>> >   3479 Fri Mar 17 05:41:16 GMT 2017
>> > org/apache/flink/hadoop/shaded/org/apache/http/params/HttpCo
>> nnectionParams.class
>> >
>> >   2392 Fri Mar 17 05:41:24 GMT 2017
>> > org/apache/commons/httpclient/params/HttpConnectionParams.class
>> >
>> >   2868 Fri Mar 17 05:41:24 GMT 2017
>> > org/apache/http/params/HttpConnectionParams.class
>> >
>> >
>> >
>> > And, yes, this is after going into flink-dist and running mvn clean
>> install
>> > again since I am using Maven 3.3.x.
>> >
>> >
>> >
>> > Here’s a snippet from my Maven version:
>> >
>> > mvn -version
>> >
>> > Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
>> > 2015-11-10T16:41:47+00:00)
>> >
>> > Maven home: /usr/local/apache-maven
>> >
>> > Java version: 1.8.0_121, vendor: Oracle Corporation
>> >
>> >
>> >
>> > Any ideas on what my problem might be here?
>> >
>> >
>> >
>> > Thanks,
>> >
>> > Craig
>> >
>> >
>>
>>
>>
>>
>

Mime
View raw message