hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@hortonworks.com>
Subject Re: Plans of moving towards JDK7 in trunk
Date Sat, 05 Apr 2014 17:52:29 GMT
On 5 April 2014 11:53, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:

> I've been using JDK7 for Hadoop development for a while now, and I
> know a lot of other folks have as well.  Correct me if I'm wrong, but
> what we're talking about here is not "moving towards JDK7" but
> "breaking compatibility with JDK6."


> There are a lot of good reasons to ditch JDK6.  It would let us use
> new APIs in JDK7, especially the new file APIs.  It would let us
> update a few dependencies to newer versions.

> I don't like the idea of breaking compatibility with JDK6 in trunk,
> but not in branch-2.  The traditional reason for putting something in
> trunk but not in branch-2 is that it is new code that needs some time
> to prove itself.

+1. branch-2 must continue to run on JDK6

> This doesn't really apply to incrementing min.jdk--
> we could do that easily whenever we like.  Meanwhile, if trunk starts
> accumulating jdk7-only code and dependencies, backports from trunk to
> branch-2 will become harder and harder over time.

I agree, but note that trunk diverges from branch-2 over time anyway -it's

> Since we make stable releases off of branch-2, and not trunk, I don't
> see any upside to this.  To be honest, I see only negatives here.
> More time backporting, more issues that show up only in production
> (branch-2) and not on dev machines (trunk).

> Maybe it's time to start thinking about what version of branch-2 will
> drop jdk6 support.  But until there is such a version, I don't think
> trunk should do it.

   1. Let's assume that branch-2 will never drop JDK6 -clusters are
   committed to it, and saying "JDK updated needed" will simply stop updates.
   2. By the hadoop 3.0 ships -2015?- JDK6 will be EOL, java 8 will be in
   common use, and even JDK7 seen as trailing edge.
   3. JDK7  improves JVM performance: NUMA, nativeIO &c -which you get for
   free -as we're confident its stable there's no reason to not move to it in
   4. As we update the dependencies on hadoop 3, we'll end up upgrading to
   libraries that are JDK7+ only (jetty!), so JDK6 is implicitly abandoned.
   5. There are new packages and APIs in Java7 which we can adopt to make
   our lives better and development more productive -as well as improving the
   user experience.

as a case in point, java.io.File.mkdirs() says "true if and only if the
directory was created; false otherwise " , and returns false in either of
the two cases:
 -the path resolves to a directory that exists
 -the path resolves to a file
think about that, anyone using local filesystems could write code that
assumes that mkdir()==0 is harmless, because if you apply it more than once
on a directory it is. But call it on a file and you don't get told its only
a file until you try to do something under it, and then things stop

In comparison, java.nio.files.Files differentiates this case by declaring
"FileAlreadyExistsException - if dir exists but is not a directory". Which
is the kind of thing that would make RawLocalFS behave a lot more like
HDFS. Similarly, if we could switch to Files.moveTo(), then the destination
file would stop being overwritten if it existed, so RawLocalFS's rename()
semantics would come closer to HDFS.

These are things we just can't do while retaining Java 6 compatibility.
-and why I am looking forward to the time when we can stop caring about

Now, assuming that Hadoop 3.x will be Java7+ only, we have the option
between now and its future ship date to move to those Java7 APIs. So when
to make the move?

   1. It can be done late -in which case few changes will happen, nobody
   sees much benefit.
   2. We can do it now, and have 12+ months to adopt the new features, make
   the move -and be set up for Java 8 migration in later versions.

Yes, code that uses the new APIs won't work on Java6, but that doesn't mean
it shouldn't happen Hadoop made the jump from Java 5 to Java 6 after all.


NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message