flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: [DISCUSS] Create a shaded Hadoop fat jar to resolve library version conflicts
Date Thu, 19 Feb 2015 09:09:18 GMT
Per offline discussion with Robert: He or I will get onto this, probably
next week.

On Thu, Feb 19, 2015 at 10:07 AM, Till Rohrmann <trohrmann@apache.org>
wrote:

> Dependency conflicts were also the reason why we have to use a different
> Akka version for the Hadoop 2.0.0-alpha build profile.
>
> Thus, +1.
>
> On Wed, Feb 18, 2015 at 3:48 PM, Robert Metzger <rmetzger@apache.org>
> wrote:
>
> > I'm also in favor of shading commonly used libraries to resolve this
> issue
> > for our upstream users.
> >
> > I recently wrote this distributed TPC-H datagenerator, which had a hard
> > dependency on a newer guava version. So I needed to shade guava in my
> > project to make it work.
> > Another candidate to shade is the ASM library.
> >
> > So you have my +1 for doing this.
> >
> > On Tue, Feb 17, 2015 at 4:26 PM, Stephan Ewen <sewen@apache.org> wrote:
> >
> > > I think the way we shaded Guava is a problem for the way IntelliJ uses
> > > maven (compile dependency projects, not package them).
> > >
> > > Since we do not apply relocation to our code for this, it should have
> no
> > > effect on the IDE usability.
> > >
> > > On Tue, Feb 17, 2015 at 3:37 PM, Ufuk Celebi <uce@apache.org> wrote:
> > >
> > > >
> > > > On 17 Feb 2015, at 09:40, Stephan Ewen <sewen@apache.org> wrote:
> > > >
> > > > > Hi everyone!
> > > > >
> > > > > We have been time and time again struck by the problem that Hadoop
> > > > bundles
> > > > > many dependencies in certain versions, that conflict either with
> > > versions
> > > > > of the dependencies we use, or with versions that users use.
> > > > >
> > > > > The most prominent examples are Guava and Protobuf.
> > > > >
> > > > > One way to solve this is to create ourselves a "custom" Hadoop
> > > > dependency:
> > > > > We build a fat jar from Hadoop (with all its dependencies) and in
> > this
> > > > fat
> > > > > jar shade Guava and Protobuf.
> > > > >
> > > > > We can use the flink-shading project for this - it is already used
> to
> > > > build
> > > > > a custom (shaded) version of Guava for Flink.
> > > > >
> > > > > Any opinions on this?
> > > >
> > > > I'm not familiar with all the details, but from past experience I
> > recall
> > > > that this has been an issue for various users. I don't see all
> > > implications
> > > > of such a change (are there any down sides?), but in general I think
> > that
> > > > it will improve the overall user experience. Hence: +1.
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message