Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
MIME-Version: 1.0
In-Reply-To: <CANC1h_tTwSgQW1cg4HerXPOdckZYMDB3ojQmc9Q7YngrSr-kVg@mail.gmail.com>
References: <CANC1h_vVL2zumbDqTbZ+pCC3unsLWOfgMaw5cSSwiofBRohNLA@mail.gmail.com>
 <997c685c-d95b-b796-a183-a22d6d384946@apache.org> <CANC1h_tTwSgQW1cg4HerXPOdckZYMDB3ojQmc9Q7YngrSr-kVg@mail.gmail.com>
From: Ufuk Celebi <uce@apache.org>
Date: Thu, 11 May 2017 10:52:25 +0200
Message-ID: <CAKiyyaH_hysb9knF2WfW8aSFj3nS=vzOC=797Gq-UGuTT6EOpA@mail.gmail.com>
Subject: Re: [DISCUSS] Changing Flink's shading model
To: dev@flink.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
archived-at: Thu, 11 May 2017 08:53:16 -0000

The advantages you've listed sound really compelling to me.

- Do you have time to implement these changes or do we need a volunteer? ;)

- I assume that republishing the artifacts as you propose doesn't have
any new legal implications since we already publish them with our
JARs, right?

- We might think about adding Netty to the list of shaded artifacts
since some dependency conflicts were reported recently. Would have to
double check the reported issues before doing that though. ;-)

=E2=80=93 Ufuk


On Wed, May 10, 2017 at 8:45 PM, Stephan Ewen <sewen@apache.org> wrote:
> @chesnay: I used ASM as an example in the proposal. Maybe I did not say
> that clearly.
>
> If we like that approach, we should deal with the other libraries (at lea=
st
> the frequently used ones) in the same way.
>
>
> I would imagine to have a project layout like that:
>
> flink-shaded-deps
>   - flink-shaded-asm
>   - flink-shaded-guava
>   - flink-shaded-curator
>   - flink-shaded-hadoop
>
>
> "flink-shaded-deps" would not be built every time (and not be released
> every time), but only when needed.
>
>
>
>
>
>
> On Wed, May 10, 2017 at 7:28 PM, Chesnay Schepler <chesnay@apache.org>
> wrote:
>
>> I like the idea, thank you for bringing it up.
>>
>> Given that the raised problems aren't really ASM specific would it make
>> sense to create one flink-shaded module that contains all frequently sha=
ded
>> libraries? (or maybe even all shaded dependencies by core modules) The
>> proposal limits the scope of this to ASM and i was wondering why.
>>
>> I also remember that there was a discussion recently about why we shade
>> things at all, and the idea of working against the shaded namespaces was
>> brought up. Back then i was expressing doubts as to whether IDE's would
>> properly support this; what's the state on that?
>>
>> On 10.05.2017 18:18, Stephan Ewen wrote:
>>
>>> Hi!
>>>
>>> This is a discussion about altering the way we handle dependencies and
>>> shading in Flink.
>>> I ran into quite a view problems trying to adjust / fix some shading
>>> issues
>>> during release validation.
>>>
>>> The issue is tracked under: https://issues.apache.org/jira
>>> /browse/FLINK-6529
>>> Bring this discussion thread up because it is a bigger issue
>>>
>>> *Problem*
>>>
>>> Currently, Flink shades dependencies like ASM and Guava into all jars o=
f
>>> projects that reference it and relocate the classes.
>>>
>>> There are some drawbacks to that approach, let's discuss them at the
>>> example of ASM:
>>>
>>>    - The ASM classes are for example in flink-core, flink-java,
>>> flink-scala,
>>> flink-runtime, etc.
>>>
>>>    - Users that reference these dependencies have the classes multiple
>>> times
>>> in the classpath. That is unclean (works, through, because the classes =
are
>>> identical). The same happens when building the final dist. jar.
>>>
>>>    - Some of these dependencies require to include license files in the
>>> shaded jar. It is hard to impossible to build a good automatic solution
>>> for
>>> that, partly due to Maven's very poor cross-project path support
>>>
>>>    - Most importantly: Scala does not support shading really well. Scal=
a
>>> classes have references to classes in more places than just the class
>>> names
>>> (apparently for Scala reflect support). Referencing a Scala project wit=
h
>>> shaded ASM still requires to add a reference to unshaded ASM (at least =
as
>>> a
>>> compile dependency).
>>>
>>> *Proposal*
>>>
>>> I propose that we build and deploy a asm-flink-shaded version of ASM an=
d
>>> directly program against the relocated namespaces. Since we never use
>>> classes that we relocate in public interfaces, Flink users will never s=
ee
>>> the relocated class names. Internally, it does not hurt to use them.
>>>
>>>    - Proper maven dependency management, no hidden (shaded) dependencie=
s
>>>
>>>    - One copy of each class for shaded dependencies
>>>
>>>    - Proper Scala interoperability
>>>
>>>    - Natural License management (license is part of deployed
>>> asm-flink-shaded jar)
>>>
>>>
>>> Happy to hear thoughts!
>>>
>>> Stephan
>>>
>>>
>>