spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yin Huai (JIRA)" <>
Subject [jira] [Commented] (SPARK-7819) Isolated Hive Client Loader appears to cause Native Library already loaded in another classloader error
Date Fri, 29 May 2015 00:23:17 GMT


Yin Huai commented on SPARK-7819:

[~coderfi] I just checked in a bug fix related to the class loader and the spark sql conf
set in spark conf (e.g. spark-default). Can you try to the latest 1.4 branch and put the following
entry in the {{conf/spark-defaults.conf}} 

{{spark.sql.hive.metastore.sharedPrefixes com.mysql.jdbc,org.postgresql,,oracle.jdbc,com.mapr.fs.shim.LibraryLoader,,com.mapr.fs.jni}}

Basically, it contains a few packages for JDBC drivers and a few mapr packages.

We will try to figure out a way to let JNI libs work with our two classloaders.

> Isolated Hive Client Loader appears to cause Native Library
already loaded in another classloader error
> -----------------------------------------------------------------------------------------------------------------------------------
>                 Key: SPARK-7819
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.4.0
>            Reporter: Fi
>            Priority: Critical
>         Attachments: stacktrace.txt,
> In reference to the pull request:
> I have been running the Spark 1.3 branch for some time with no major hiccups, and recently
switched to the Spark 1.4 branch.
> I build my spark distribution with the following build command:
> {noformat}
> --tgz --skip-java-test --with-tachyon -Phive -Phive-0.13.1 -Pmapr4
-Pspark-ganglia-lgpl -Pkinesis-asl -Phive-thriftserver
> {noformat}
> When running a python script containing a series of smoke tests I use to validate the
build, I encountered an error under the following conditions:
> * start a spark context
> * start a hive context
> * run any hive query
> * stop the spark context
> * start a second spark context
> * run any hive query
> ** ERROR
> From what I can tell, the Isolated Class Loader is hitting a MapR class that is loading
its native library (presumedly as part of a static initializer).
> Unfortunately, the JVM prohibits this the second time around.
> I would think that shutting down the SparkContext would clear out any vestigials of the
JVM, so I'm surprised that this would even be a problem.
> Note: all other smoke tests we are running passes fine.
> I will attach the stacktrace and a python script reproducing the issue (at least for
my environment and build).

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message