spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jungtaek Lim (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-11818) REPL ExecutorClassLoader cannot see any resources from parent class loader
Date Wed, 18 Nov 2015 15:20:11 GMT

    [ https://issues.apache.org/jira/browse/SPARK-11818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15011168#comment-15011168
] 

Jungtaek Lim edited comment on SPARK-11818 at 11/18/15 3:19 PM:
----------------------------------------------------------------

Yes, assembly jar has HBase libs, and unzipped assembly jar has hbase-default.xml (because
it is one of files in hbase-common lib).

I attached assembly jar by --jars options, and run some codes which initializes HBase configuration
from "Executor side".
Executor found "spark.repl.class.uri" config, and replace current thread's context class loader
to ExecutorClassLoader, which has URLClassLoader to parent class loader.

Assembly jar is loaded to parent class loader via Executor.updateDependencies(), and ExecutorClassLoader
can see classes from parent class loader so there're no issue to load classes.
But HBaseConfiguration.addHbaseResources() tries to load hbase-default.xml from classloader
which seems not succeed.


was (Author: kabhwan):
Yes, assembly jar has HBase libs, and unzipped assembly jar has hbase-default.xml.

I attached assembly jar by --jars options, and run some codes which initializes HBase configuration
from "Executor side".
Executor found "spark.repl.class.uri" config, and replace current thread's context class loader
to ExecutorClassLoader, which has URLClassLoader to parent class loader.

Assembly jar is loaded to parent class loader via Executor.updateDependencies(), and ExecutorClassLoader
can see classes from parent class loader so there're no issue to load classes.
But HBaseConfiguration.addHbaseResources() tries to load hbase-default.xml from classloader
which seems not succeed.

> REPL ExecutorClassLoader cannot see any resources from parent class loader
> --------------------------------------------------------------------------
>
>                 Key: SPARK-11818
>                 URL: https://issues.apache.org/jira/browse/SPARK-11818
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Shell
>         Environment: CentOS 6, spark 1.4.1-hadoop2.4, mesos 0.22.1
>            Reporter: Jungtaek Lim
>
> This issue starts from finding root reason from strange problem from spark-shell (and
zeppelin) which is not a problem for spark-submit.
> https://mail-archives.apache.org/mod_mbox/spark-user/201511.mbox/%3CCAF5108jMXyOjiGmCgr%3Ds%2BNvTMcyKWMBVM1GsrH7Pz4xUj48LfA%40mail.gmail.com%3E
> After some hours (over days) digging into the detail, I found that ExecutorClassLoader
cannot see any resource files which can be seen from parent class loader.
> ExecutorClassLoader itself doesn't need to lookup resource files cause REPL doesn't generate
these, but it should delegate lookup to parent class loader.
> I'll provide the pull request which includes tests which fails on master soon.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message