hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@hortonworks.com>
Subject Re: Moving to JDK7, JDK8 and new major releases
Date Tue, 24 Jun 2014 22:35:02 GMT
That classpath policy was explicitly added because we can't lock down our
dependencies for security/bug fix reasons, and also because if we do update
something explicitly, their transitive dependencies can change -beyond our
control.

https://issues.apache.org/jira/browse/HADOOP-9555 is an example of this: an
update of ZK explicitly to fix an HA problem. Are there changes in its
dependencies? I don't know. But we didn't have a choice to update if we
wanted NN & RM failover to work reliably, so we have to take any other
changes that went in.

JDK upgrades can be viewed as an extension of this -we are changing the
base platform that Hadoop runs on. More precisely, for the Java 6- >Java 7
update, we are reflecting the fact that nobody is running in production on
Java 6

Do you realise we actually moved to Java 6 in 2008?
https://issues.apache.org/jira/browse/HADOOP-2325 . That was six years ago
-half the names on that list are not active on the project any more.

What we did there was issue a warning in 0.18 that it would be the last
Java 5 version; 0.19  moved up -we can do the same for a Hadoop 2.x release
at some point this year.



On 24 June 2014 11:43, Arun C Murthy <acm@hortonworks.com> wrote:

> Andrew,
>
>  Thanks for starting this thread. I'll edit the wiki to provide more
> context around rolling-upgrades etc. which, as I pointed out in the
> original thread, are key IMHO.
>
> On Jun 24, 2014, at 11:17 AM, Andrew Wang <andrew.wang@cloudera.com>
> wrote:
> > https://wiki.apache.org/hadoop/MovingToJdk7and8
> >
> > I think based on our current compatibility guidelines, Proposal A is the
> > most attractive. We're pretty hamstrung by the requirement to keep the
> > classpath the same, which would be solved by either OSGI or shading our
> > deps (but that's a different discussion).
>
> I don't see that anywhere in our current compatibility guidelines.
>
> As you can see from
> http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/Compatibility.html
> we do not have such a policy (pasted here for convenience):
>
> Java Classpath
>
> User applications built against Hadoop might add all Hadoop jars
> (including Hadoop's library dependencies) to the application's classpath.
> Adding new dependencies or updating the version of existing dependencies
> may interfere with those in applications' classpaths.
>
> Policy
>
> Currently, there is NO policy on when Hadoop's dependencies can change.
>
> Furthermore, we have *already* changed our classpath in hadoop-2.x. Again,
> as I pointed out in the previous thread, here is the precedent:
>
> On Jun 21, 2014, at 5:59 PM, Arun C Murthy <acm@hortonworks.com> wrote:
>
> > Also, this is something we already have done i.e. we updated some of our
> software deps in hadoop-2.4 v/s hadoop-2.2 - clearly not something as
> dramatic as JDK. Here are some examples:
> > https://issues.apache.org/jira/browse/HADOOP-9991
> > https://issues.apache.org/jira/browse/HADOOP-10102
> > https://issues.apache.org/jira/browse/HADOOP-10103
> > https://issues.apache.org/jira/browse/HADOOP-10104
> > https://issues.apache.org/jira/browse/HADOOP-10503
>
> thanks,
> Arun
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message