spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shixiong Zhu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-20547) ExecutorClassLoader's findClass may not work correctly when a task is cancelled.
Date Tue, 02 May 2017 18:10:04 GMT

    [ https://issues.apache.org/jira/browse/SPARK-20547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15993426#comment-15993426
] 

Shixiong Zhu commented on SPARK-20547:
--------------------------------------

Did some investigation using the reproducer. Looks like it’s not a class loader issue. It’s
because the class initialization result will be cached. My current proposal is recreating
a new class loader if a task fails.

> ExecutorClassLoader's findClass may not work correctly when a task is cancelled.
> --------------------------------------------------------------------------------
>
>                 Key: SPARK-20547
>                 URL: https://issues.apache.org/jira/browse/SPARK-20547
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Shell
>    Affects Versions: 2.1.0, 2.2.0
>            Reporter: Shixiong Zhu
>            Priority: Blocker
>
> ExecutorClassLoader's findClass may throw some transient exception. For example, when
a task is cancelled, if ExecutorClassLoader is running, you may see InterruptedException or
IOException, even if this class can be loaded. Then the result of findClass will be cached
by JVM, and later when the same class is being loaded (note: in this case, this class may
be still loadable), it will just throw NoClassDefFoundError.
> We should make ExecutorClassLoader retry on transient exceptions.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message