flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: [DISCUSS] Create a shaded Hadoop fat jar to resolve library version conflicts
Date Tue, 17 Feb 2015 15:26:09 GMT
I think the way we shaded Guava is a problem for the way IntelliJ uses
maven (compile dependency projects, not package them).

Since we do not apply relocation to our code for this, it should have no
effect on the IDE usability.

On Tue, Feb 17, 2015 at 3:37 PM, Ufuk Celebi <uce@apache.org> wrote:

>
> On 17 Feb 2015, at 09:40, Stephan Ewen <sewen@apache.org> wrote:
>
> > Hi everyone!
> >
> > We have been time and time again struck by the problem that Hadoop
> bundles
> > many dependencies in certain versions, that conflict either with versions
> > of the dependencies we use, or with versions that users use.
> >
> > The most prominent examples are Guava and Protobuf.
> >
> > One way to solve this is to create ourselves a "custom" Hadoop
> dependency:
> > We build a fat jar from Hadoop (with all its dependencies) and in this
> fat
> > jar shade Guava and Protobuf.
> >
> > We can use the flink-shading project for this - it is already used to
> build
> > a custom (shaded) version of Guava for Flink.
> >
> > Any opinions on this?
>
> I'm not familiar with all the details, but from past experience I recall
> that this has been an issue for various users. I don't see all implications
> of such a change (are there any down sides?), but in general I think that
> it will improve the overall user experience. Hence: +1.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message