hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@hortonworks.com>
Subject Re: Time to address the Guava version problem
Date Mon, 22 Sep 2014 08:08:40 GMT
On 21 September 2014 23:11, Karthik Kambatla <kasha@cloudera.com> wrote:

> Upgrading Guava version is tricky. While it helps in many cases, it can
> break existing applications/deployments. I understand we do not have a
> policy for updating dependencies, but still we should be careful with
> Guava.
>

I agree, but the classpath is currently in an inconsistent state: it
incudes Guava 11 and a library  built against Guava 16.


>
> I would be more inclined towards a more permanent solution to this problem
> - how about prioritizing classpath isolation so applications aren't
> affected by Hadoop dependency updates at all? I understand that will also
> break user applications, but it might be the driving feature for Hadoop
> 3.0?
>


I think this would be good;

if you look at where we're going with YARN-deployed apps, there is a trend
towards pushing up all the JARs, using the distributed cache to reduce the
cost of that upload. All you should really need at the far end are the .xml
files.

Except this has caused a problem with branch-2 to surface, the native libs
aren't binary-signature-compatible with 2.5 or earlier JARs -look  at
HADOOP-11064

This something to be fixed —but it highlights the problem where even the
native .lib, .so. .dll files are implicitly part of the in-hadoop
compatibility layer.

so: the "upload all JARs" strategy has weaknesses too; some OSGi solution
could address that, though we need time playing with that before being able
to claim it solves all problems ... I worry that it may help address JAR
dependencies at the price of performance.



Returning to Guava, which can't be put off as we are already in a mess


>
> On Fri, Sep 19, 2014 at 5:13 PM, Sangjin Lee <sjlee@apache.org> wrote:
>
> > I would also agree on upgrading guava. Yes I am aware of the potential
> > impact on customers who might rely on hadoop bringing in guava 11.
> However,
> > IMHO the balance tipped over to the other side a while ago; i.e. I think
> > there are far more people using guava 16 in their code and scrambling to
> > make things work than the other way around.
> >
>

I concur ... to many things you build downstream need a 15+ guava. Which
includes Hadoop now ... we just haven't fully admitted it yet.

Steve






> > On Thu, Sep 18, 2014 at 2:40 PM, Steve Loughran <stevel@hortonworks.com>
> > wrote:
> >
> > > I know we've been ignoring the Guava version problem, but HADOOP-10868
> > > added a transitive dependency on Guava 16 by way of Curator 2.6.
> > >
> > > Maven currently forces the build to use Guava 11.0.2, but this is
> hiding
> > at
> > > compile timeall code paths from curator which may use classes & methods
> > > that aren't there.
> > >
> > > I need curator for my own work (2.4.1 & Guava 14.0 was what I'd been
> > > using), so don't think we can go back.
> > >
> > > HADOOP-11102 covers the problem -but doesn't propose a specific
> solution.
> > > But to me the one that seems most likely to work is: update Guava
> > >
> > > -steve
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message