From issues-return-217320-archive-asf-public=cust-asf.ponee.io@spark.apache.org Fri Mar 1 17:35:14 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 749FB180647 for ; Fri, 1 Mar 2019 18:35:13 +0100 (CET) Received: (qmail 96956 invoked by uid 500); 1 Mar 2019 17:35:12 -0000 Mailing-List: contact issues-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@spark.apache.org Received: (qmail 96947 invoked by uid 99); 1 Mar 2019 17:35:12 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Mar 2019 17:35:12 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 0B2CC180D39 for ; Fri, 1 Mar 2019 17:35:12 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.5 X-Spam-Level: X-Spam-Status: No, score=-109.5 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id jctSpdhg5czn for ; Fri, 1 Mar 2019 17:35:10 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 1BA1C5F56B for ; Fri, 1 Mar 2019 17:35:10 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 932DDE280B for ; Fri, 1 Mar 2019 17:35:03 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id AC56025707 for ; Fri, 1 Mar 2019 17:35:00 +0000 (UTC) Date: Fri, 1 Mar 2019 17:35:00 +0000 (UTC) From: "Sean Owen (JIRA)" To: issues@spark.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (SPARK-26839) on JDK11, IsolatedClientLoader must be able to load java.sql classes MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/SPARK-26839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16781903#comment-16781903 ] Sean Owen commented on SPARK-26839: ----------------------------------- Just sharing where I have gotten to with this as well: I think we need to update IsolatedClientLoader to use getPlatformClassLoader() on Java 9+ as the parent. Right now it uses getSystemClassloader().getParent().getParent(), which is the boot classloader, which is actually always represented by null. (There's an assert in the code that checks if loading Hive classes fails, and it always fails due to an NPE! That part is easy. The problem here is actually datanucleus, which needs to be 5.1+ for Java 9, it seems. I was uneasy about updating from 3.2 because Hive uses it, but it seems to get past the immediate error here. I have a PR that does both of these things. However it now fails for not finding BONECP, which is the pool I think Hive uses. It's on the classpath though. I suspect it's something to do with how we try to isolate Hive-related classes. I put BONECP on the classpath explicitly and still no dice. That's where I got so far. > on JDK11, IsolatedClientLoader must be able to load java.sql classes > -------------------------------------------------------------------- > > Key: SPARK-26839 > URL: https://issues.apache.org/jira/browse/SPARK-26839 > Project: Spark > Issue Type: Sub-task > Components: SQL > Affects Versions: 3.0.0 > Reporter: Imran Rashid > Priority: Major > > This might be very specific to my fork & a kind of weird system setup I'm working on, I haven't completely confirmed yet, but I wanted to report it anyway in case anybody else sees this. > When I try to do anything which touches the metastore on java11, I immediately get errors from IsolatedClientLoader that it can't load anything in java.sql. eg. > {noformat} > scala> spark.sql("show tables").show() > java.lang.ClassNotFoundException: java.lang.NoClassDefFoundError: java/sql/SQLTransientException when creating Hive client using classpath: file:/home/systest/jdk-11.0.2/, ... > ... > Caused by: java.lang.ClassNotFoundException: java.sql.SQLTransientException > at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471) > at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588) > at org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1.doLoadClass(IsolatedClientLoader.scala:230) > at org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1.loadClass(IsolatedClientLoader.scala:219) > at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521) > {noformat} > After a bit of debugging, I also discovered that the {{rootClassLoader}} is {{null}} in {{IsolatedClientLoader}}. I think this would work if either {{rootClassLoader}} could load those classes, or if {{isShared()}} was changed to allow any class starting with "java." (I'm not sure why it only allows "java.lang" and "java.net" currently.) -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org For additional commands, e-mail: issues-help@spark.apache.org