spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <>
Subject Re: Anyone knows how to build and spark on jdk9?
Date Mon, 30 Oct 2017 14:51:11 GMT

On 27 Oct 2017, at 19:24, Sean Owen <<>>

Certainly, Scala 2.12 support precedes Java 9 support. A lot of the work is in place already,
and the last issue is dealing with how Scala closures are now implemented quite different
with lambdas / invokedynamic. This affects the ClosureCleaner. For the interested, this is
as far as I know the main remaining issue:

Despite the odd naming, all of these versions of Java are successors to Java 9. Supporting
any of them is probably the same thing, so, the work is still for now getting it working on
Java 9.

Whereas Java has been very backwards-compatible in the past, the new module structure is almost
certain to break something in Spark or its dependencies. Removing JAXB from the JDK alone
causes issues. Getting it to run at all on Java 9 may require changes, whereas compatibility
with new Java major releases in the past generally came for free. It'll be worth trying to
make that happen soonish. I'm guessing for Spark 3.x in first half of next year?

it is going to be traumatic across the stack, but it's probably best starting it as a background
activity, just to be aware of what's going to work and where the trouble is(*).

But, first things first. Scala 2.12 support.

On Fri, Oct 27, 2017 at 6:02 PM Jörn Franke <<>>
Scala 2.12 is not yet supported on Spark - this means also not JDK9:

If you look at the Oracle support then jdk 9 is anyway only supported for 6 months. JDK 8
is Lts (5 years) JDK 18.3 will be only 6 months and JDK 18.9 is lts (5 years).

I do not think Spark should support non-lts releases. Especially for JDK9 I do not see a strong
technical need, but maybe I am overlooking something. Of course http2 etc would be nice for
the web interfaces, but currently not very urgent.

Oracle's new retirement strategy is "odd": it'll essentially be killing java 0 updates before
java 8, and retiring java 8 as the same time as the march '18 release. Like you say, not very
motiviational for an update.

At the same time: Java 8 is going away, and at some point the move to the new versions will
be needed, even if the new version isn't JDK9 itself. It's generally helpful to be a bit proactive,
especially getting all the dependencies bumped up, sorting out build & test. The real
enemy is any incompatible change needed in the code, or something which breaks public/stable
APIs. That and some dependency on a library which is not compatible with java 9 and which
lacks a replacement. Either you take on the maintenance yourself (bad),  or you do the migration.

(*) I predict "Kerberos". it's always Kerberos. A move to a "per-app JRE will complicate enabling
full length bit encryption", as the ASF isn't going to be able to ship the extended crypto
JAR needed for Kerberos ad 256 bit keys.

View raw message