hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Wang <andrew.w...@cloudera.com>
Subject Re: very old dependencies
Date Sat, 29 Mar 2014 02:41:41 GMT
We've successfully supported CDH4.2+ on Java 7 as well as CDH5, so I think
that code wise we're ready to move to 7. However, we can't start using Java
7-only features in the 2.x line for compatibility reasons.

Also, even if the compatibility guidelines state that we can bump our
dependencies whenever we want, practically speaking we can't in the 2.x
line without breaking existing applications. I think the best we can do is
fix this for 3.x by shading everything so we don't hit this issue again.

Andrew


On Fri, Mar 28, 2014 at 5:34 PM, Colin McCabe <cmccabe@alumni.cmu.edu>wrote:

> I think we need some way of isolating YARN, MR, and HDFS clients from
> the Hadoop dependencies.  Anything else just isn't sane... whatever we
> may say, there will always be clients that rely on the dependencies
> that we pull in, if we make those visible.  I can't really blame
> clients for this.  It's sort of unclear what happens when you pull in
> multiple versions of a dependency through Maven.
>
> The easiest way to solve this would be to use something like Maven-shade.
>
> best,
> Colin
>
> On Fri, Mar 28, 2014 at 10:30 AM, Steve Loughran <stevel@hortonworks.com>
> wrote:
> > I don't disagree about version age -I've just been downgrading a project
> to
> > use some of the versions. The issue with YARN apps is that you get these
> on
> > your classpath.
> >
> >
> >    1. There's a JIRA: https://issues.apache.org/jira/browse/HADOOP-9991
> >    2. Nobody is working full time on these, I sporadically apply the
> >    patches -I get to feel the pain downstream
> >
> > commons lang is at 2.6 now, which should keep you happy.
> >
> > jetty? Now that the shuffle is on netty we could think about this and
> jersey
> >
> > Guava is trouble. Leave it: new code doesn't work. Remove it and old code
> > stops working.
> > https://issues.apache.org/jira/browse/HADOOP-10101
> >
> > I think for branch-3 we should go ahead an apply the patch. For branch 2,
> > it's too destructuve.
> >
> > At some point we also have to commit being java7 + only -this would
> > actually help us move up to some of the later dependencies. That's
> clearly
> > not a branch-2
> >
> >
> >
> >
> > On 28 March 2014 14:59, Sangjin Lee <sjlee0@gmail.com> wrote:
> >
> >> Hi folks,
> >>
> >> Even as 2.3 was released, several dependencies of hadoop are quite
> dated.
> >> And more and more of them are causing friction for hadoop-related
> libraries
> >> and hadoop users in general, as these dependency versions are often
> >> incompatibly different than the versions most people use these days. So
> >> this is becoming a very real problem, if not one already.
> >>
> >> Some key ones include
> >> - guava: 11.0.2 (current 16.0.1)
> >> - jetty: 6.1.26 (current 9.1.3)
> >> - commons-lang: 2.6 (current 3.3.1)
> >>
> >> In particular, guava is already causing a lot of issues as many
> developers
> >> have moved onto a newer version and are using APIs that do not exist in
> >> 11.0.2, and get NoSuchMethodErrors and such.
> >>
> >> Also, for jetty, 6.1.26 has been EOFed for some time now.
> >>
> >> Do we have a plan to review some of these dependencies and upgrade them
> to
> >> reasonably up-to-date versions? I remember seeing some JIRAs on specific
> >> dependencies, but has a review been done to determine a good set of
> >> versions to upgrade to?
> >>
> >> It would be great if we could upgrade some of the more common ones at
> least
> >> to modern versions.
> >>
> >> Thanks,
> >> Sangjin
> >>
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message