spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <>
Subject [jira] [Commented] (SPARK-26839) on JDK11, IsolatedClientLoader must be able to load java.sql classes
Date Fri, 01 Mar 2019 17:35:00 GMT


Sean Owen commented on SPARK-26839:

Just sharing where I have gotten to with this as well:
I think we need to update IsolatedClientLoader to use getPlatformClassLoader() on Java 9+
as the parent. Right now it uses getSystemClassloader().getParent().getParent(), which is
the boot classloader, which is actually always represented by null. (There's an assert in
the code that checks if loading Hive classes fails, and it always fails due to an NPE! That
part is easy.

The problem here is actually datanucleus, which needs to be 5.1+ for Java 9, it seems. I was
uneasy about updating from 3.2 because Hive uses it, but it seems to get past the immediate
error here. I have a PR that does both of these things.

However it now fails for not finding BONECP, which is the pool I think Hive uses. It's on
the classpath though. I suspect it's something to do with how we try to isolate Hive-related
classes. I put BONECP on the classpath explicitly and still no dice. That's where I got so

> on JDK11, IsolatedClientLoader must be able to load java.sql classes
> --------------------------------------------------------------------
>                 Key: SPARK-26839
>                 URL:
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>    Affects Versions: 3.0.0
>            Reporter: Imran Rashid
>            Priority: Major
> This might be very specific to my fork & a kind of weird system setup I'm working
on, I haven't completely confirmed yet, but I wanted to report it anyway in case anybody else
sees this.
> When I try to do anything which touches the metastore on java11, I immediately get errors
from IsolatedClientLoader that it can't load anything in java.sql.  eg.
> {noformat}
> scala> spark.sql("show tables").show()
> java.lang.ClassNotFoundException: java.lang.NoClassDefFoundError: java/sql/SQLTransientException
when creating Hive client using classpath: file:/home/systest/jdk-11.0.2/, ...
> ...
> Caused by: java.lang.ClassNotFoundException: java.sql.SQLTransientException
>   at java.base/
>   at java.base/java.lang.ClassLoader.loadClass(
>   at org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1.doLoadClass(IsolatedClientLoader.scala:230)
>   at org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1.loadClass(IsolatedClientLoader.scala:219)
>   at java.base/java.lang.ClassLoader.loadClass(
> {noformat}
> After a bit of debugging, I also discovered that the {{rootClassLoader}} is {{null}}
in {{IsolatedClientLoader}}.  I think this would work if either {{rootClassLoader}} could
load those classes, or if {{isShared()}} was changed to allow any class starting with "java."
 (I'm not sure why it only allows "java.lang" and "" currently.)

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message