flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Till Rohrmann <trohrm...@apache.org>
Subject Re: [DISCUSS] Create a shaded Hadoop fat jar to resolve library version conflicts
Date Thu, 19 Feb 2015 09:07:32 GMT
Dependency conflicts were also the reason why we have to use a different
Akka version for the Hadoop 2.0.0-alpha build profile.

Thus, +1.

On Wed, Feb 18, 2015 at 3:48 PM, Robert Metzger <rmetzger@apache.org> wrote:

> I'm also in favor of shading commonly used libraries to resolve this issue
> for our upstream users.
>
> I recently wrote this distributed TPC-H datagenerator, which had a hard
> dependency on a newer guava version. So I needed to shade guava in my
> project to make it work.
> Another candidate to shade is the ASM library.
>
> So you have my +1 for doing this.
>
> On Tue, Feb 17, 2015 at 4:26 PM, Stephan Ewen <sewen@apache.org> wrote:
>
> > I think the way we shaded Guava is a problem for the way IntelliJ uses
> > maven (compile dependency projects, not package them).
> >
> > Since we do not apply relocation to our code for this, it should have no
> > effect on the IDE usability.
> >
> > On Tue, Feb 17, 2015 at 3:37 PM, Ufuk Celebi <uce@apache.org> wrote:
> >
> > >
> > > On 17 Feb 2015, at 09:40, Stephan Ewen <sewen@apache.org> wrote:
> > >
> > > > Hi everyone!
> > > >
> > > > We have been time and time again struck by the problem that Hadoop
> > > bundles
> > > > many dependencies in certain versions, that conflict either with
> > versions
> > > > of the dependencies we use, or with versions that users use.
> > > >
> > > > The most prominent examples are Guava and Protobuf.
> > > >
> > > > One way to solve this is to create ourselves a "custom" Hadoop
> > > dependency:
> > > > We build a fat jar from Hadoop (with all its dependencies) and in
> this
> > > fat
> > > > jar shade Guava and Protobuf.
> > > >
> > > > We can use the flink-shading project for this - it is already used to
> > > build
> > > > a custom (shaded) version of Guava for Flink.
> > > >
> > > > Any opinions on this?
> > >
> > > I'm not familiar with all the details, but from past experience I
> recall
> > > that this has been an issue for various users. I don't see all
> > implications
> > > of such a change (are there any down sides?), but in general I think
> that
> > > it will improve the overall user experience. Hence: +1.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message