hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sangjin Lee <sjl...@gmail.com>
Subject Re: very old dependencies
Date Mon, 31 Mar 2014 17:56:15 GMT
I agree.

We should still be diligent in keeping dependencies up-to-date as long as
the changes are fairly non-destructive (I hear you about guava). That would
be a pretty big service to not only users but also developers of libraries
on hadoop.


On Fri, Mar 28, 2014 at 11:04 AM, Steve Loughran <stevel@hortonworks.com>wrote:

> On 28 March 2014 17:00, Sandy Ryza <sandy.ryza@cloudera.com> wrote:
>
> > My understanding is that unfortunately we're stuck with these for the
> rest
> > of 2.x, because changing them could break jobs that rely on them.  For
> jobs
> > that want to use newer versions, the recommendation is to use
> > mapreduce.user.classpath.first or turn on classpath isolation with
> > mapreduce.job.classloader.
> >
>
>
> If you look at the compatibility statement of hadoop it makes clear there
> are no guarantees about dependencies and especially transitive ones.
>
>
> http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/Compatibility.html#Java_Classpath
>
> That is not an accident -any statement of stability would risk painting us
> into a corner and never able to update anything. It also allowed us to ship
> hadoop 2.2 without doing a rush update of every dependency -that would only
> have caused chaos and delays. Protubf -> 2.5 was enough -and that was done
> because it was even worse than guava in terms of compatibility policy.
>
> The issue goes beyond MR as YARN apps pick up the core binaries, so are
> constrained by what comes in hadoop/lib. hdfs/lib and yarn/lib. most of
> which is pretty dated.
>
> and if you have an app like hbase or accumulo, with more current
> dependencies, you have to play games excluding all of hadoop's
> dependencies. But even that doesn't help with Guava, because it is so
> aggressive about retiring classes and methods.
>
> for the sake of our own code and more modern apps, I'm in favour of
> gradually rolling out updates that don't break things, force a move to
> java7 or require changes to the source.
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message