hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Billie Rinaldi <billie.rina...@gmail.com>
Subject Re: Time to address the Guava version problem
Date Wed, 24 Sep 2014 14:35:58 GMT
The use of an unnecessarily old dependency encourages problems like
HDFS-7040.  The current Guava dependency is a big problem for downstream
apps and I'd really like to see it addressed.

On Tue, Sep 23, 2014 at 2:09 PM, Steve Loughran <stevel@hortonworks.com>
wrote:

> I'm using curator elsewhere, it does log a lot (as does the ZK client), but
> it solves a lot of problem. It's being adopted more downstream too.
>
> I'm wondering if we can move the code to the extent we know it works with
> Guava 16, with the hadoop core being 16-compatible, but not actually
> migrated to 16.x only. Then hadoop ships with 16 for curator & downstream
> apps, but we say "you can probably roll back to 11 provided you don't use
> features x-y-z".
>
> On 23 September 2014 21:55, Robert Kanter <rkanter@cloudera.com> wrote:
>
> > At the same time, not being able to use Curator will require a lot of
> extra
> > code, a lot of which we probably already have from the ZKRMStateStore,
> but
> > it's not available to use in hadoop-auth.  We'd need to create our own ZK
> > libraries that Hadoop components can use, but (a) that's going to take a
> > while, and (b) it seems silly to reinvent the wheel when Curator already
> > does all this.
> >
> > I agree that upgrading Guava will be a compatibility problem though...
> >
> > On Tue, Sep 23, 2014 at 9:30 AM, Sandy Ryza <sandy.ryza@cloudera.com>
> > wrote:
> >
> > > If we've broken compatibility in branch-2, that's a bug that we need to
> > > fix. HADOOP-10868 has not yet made it into a release; I don't see it
> as a
> > > justification for solidifying the breakage.
> > >
> > > -1 to upgrading Guava in branch-2.
> > >
> > > On Tue, Sep 23, 2014 at 3:06 AM, Steve Loughran <
> stevel@hortonworks.com>
> > > wrote:
> > >
> > > > +1 to upgrading guava. Irrespective of downstream apps, the hadoop
> > source
> > > > tree is now internally inconsistent
> > > >
> > > > On 22 September 2014 17:56, Sangjin Lee <sjlee@apache.org> wrote:
> > > >
> > > > > I agree that a more robust solution is to have better classloading
> > > > > isolation.
> > > > >
> > > > > Still, IMHO guava (and possibly protobuf as well) sticks out like
a
> > > sore
> > > > > thumb. There are just too many issues in trying to support both
> guava
> > > 11
> > > > > and guava 16. Independent of what we may do with the classloading
> > > > > isolation, we should still consider upgrading guava.
> > > > >
> > > > > My 2 cents.
> > > > >
> > > > > On Sun, Sep 21, 2014 at 3:11 PM, Karthik Kambatla <
> > kasha@cloudera.com>
> > > > > wrote:
> > > > >
> > > > > > Upgrading Guava version is tricky. While it helps in many cases,
> it
> > > can
> > > > > > break existing applications/deployments. I understand we do
not
> > have
> > > a
> > > > > > policy for updating dependencies, but still we should be careful
> > with
> > > > > > Guava.
> > > > > >
> > > > > > I would be more inclined towards a more permanent solution to
> this
> > > > > problem
> > > > > > - how about prioritizing classpath isolation so applications
> aren't
> > > > > > affected by Hadoop dependency updates at all? I understand that
> > will
> > > > also
> > > > > > break user applications, but it might be the driving feature
for
> > > Hadoop
> > > > > > 3.0?
> > > > > >
> > > > > > On Fri, Sep 19, 2014 at 5:13 PM, Sangjin Lee <sjlee@apache.org>
> > > wrote:
> > > > > >
> > > > > > > I would also agree on upgrading guava. Yes I am aware of
the
> > > > potential
> > > > > > > impact on customers who might rely on hadoop bringing in
guava
> > 11.
> > > > > > However,
> > > > > > > IMHO the balance tipped over to the other side a while
ago;
> i.e.
> > I
> > > > > think
> > > > > > > there are far more people using guava 16 in their code
and
> > > scrambling
> > > > > to
> > > > > > > make things work than the other way around.
> > > > > > >
> > > > > > > On Thu, Sep 18, 2014 at 2:40 PM, Steve Loughran <
> > > > > stevel@hortonworks.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I know we've been ignoring the Guava version problem,
but
> > > > > HADOOP-10868
> > > > > > > > added a transitive dependency on Guava 16 by way of
Curator
> > 2.6.
> > > > > > > >
> > > > > > > > Maven currently forces the build to use Guava 11.0.2,
but
> this
> > is
> > > > > > hiding
> > > > > > > at
> > > > > > > > compile timeall code paths from curator which may
use
> classes &
> > > > > methods
> > > > > > > > that aren't there.
> > > > > > > >
> > > > > > > > I need curator for my own work (2.4.1 & Guava
14.0 was what
> I'd
> > > > been
> > > > > > > > using), so don't think we can go back.
> > > > > > > >
> > > > > > > > HADOOP-11102 covers the problem -but doesn't propose
a
> specific
> > > > > > solution.
> > > > > > > > But to me the one that seems most likely to work is:
update
> > Guava
> > > > > > > >
> > > > > > > > -steve
> > > > > > > >
> > > > > > > > --
> > > > > > > > CONFIDENTIALITY NOTICE
> > > > > > > > NOTICE: This message is intended for the use of the
> individual
> > or
> > > > > > entity
> > > > > > > to
> > > > > > > > which it is addressed and may contain information
that is
> > > > > confidential,
> > > > > > > > privileged and exempt from disclosure under applicable
law.
> If
> > > the
> > > > > > reader
> > > > > > > > of this message is not the intended recipient, you
are hereby
> > > > > notified
> > > > > > > that
> > > > > > > > any printing, copying, dissemination, distribution,
> disclosure
> > or
> > > > > > > > forwarding of this communication is strictly prohibited.
If
> you
> > > > have
> > > > > > > > received this communication in error, please contact
the
> sender
> > > > > > > immediately
> > > > > > > > and delete it from your system. Thank You.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or
> > entity
> > > to
> > > > which it is addressed and may contain information that is
> confidential,
> > > > privileged and exempt from disclosure under applicable law. If the
> > reader
> > > > of this message is not the intended recipient, you are hereby
> notified
> > > that
> > > > any printing, copying, dissemination, distribution, disclosure or
> > > > forwarding of this communication is strictly prohibited. If you have
> > > > received this communication in error, please contact the sender
> > > immediately
> > > > and delete it from your system. Thank You.
> > > >
> > >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message