flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ufuk Celebi <...@apache.org>
Subject Re: [DISCUSS] Create a shaded Hadoop fat jar to resolve library version conflicts
Date Tue, 17 Feb 2015 14:37:33 GMT

On 17 Feb 2015, at 09:40, Stephan Ewen <sewen@apache.org> wrote:

> Hi everyone!
> 
> We have been time and time again struck by the problem that Hadoop bundles
> many dependencies in certain versions, that conflict either with versions
> of the dependencies we use, or with versions that users use.
> 
> The most prominent examples are Guava and Protobuf.
> 
> One way to solve this is to create ourselves a "custom" Hadoop dependency:
> We build a fat jar from Hadoop (with all its dependencies) and in this fat
> jar shade Guava and Protobuf.
> 
> We can use the flink-shading project for this - it is already used to build
> a custom (shaded) version of Guava for Flink.
> 
> Any opinions on this?

I'm not familiar with all the details, but from past experience I recall that this has been
an issue for various users. I don't see all implications of such a change (are there any down
sides?), but in general I think that it will improve the overall user experience. Hence: +1.
Mime
View raw message