hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <...@hortonworks.com>
Subject Re: Moving to JDK7, JDK8 and new major releases
Date Wed, 25 Jun 2014 04:30:22 GMT
On Jun 24, 2014, at 4:22 PM, Andrew Wang <andrew.wang@cloudera.com> wrote:

> Since Hadoop apps can and do depend on the Hadoop classpath, the classpath
> is effectively part of our API. I'm sure there are user apps out there that
> will break if we make incompatible changes to the classpath. I haven't read
> up on the MR JIRA Arun mentioned, but there MR isn't the only YARN app out
> there.

I think there is a some confusion/misunderstanding here.

With hadoop-2 the user is completely in control of his own classpath (we had a similar, but
limited capability in hadoop-1 w/ https://issues.apache.org/jira/browse/MAPREDUCE-1938).

Furthermore, it's probably not well known that in hadoop-2 the user application (MR or otherwise)
can also pick the JDK version by using JAVA_HOME env for the container. So, in effect, MR
applications can continue to use java6 while YARN is running java7 - this hasn't been tested
extensively though. This capability did not exist in hadoop-1. We've also made some progress
with https://issues.apache.org/jira/browse/MAPREDUCE-1700 to defuse user jar-deps from MR
system jars. https://issues.apache.org/jira/browse/MAPREDUCE-4421 also helps by ensuring MR
applications can pick exact version of MR jars they were compiled against; and not rely on
cluster installs.

Hope that helps somewhat.


NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message