spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wenchen Fan (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-20904) Task failures during shutdown cause problems with preempted executors
Date Sun, 23 Jul 2017 15:25:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-20904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Wenchen Fan resolved SPARK-20904.
---------------------------------
       Resolution: Fixed
    Fix Version/s: 2.3.0
                   2.2.1

Issue resolved by pull request 18594
[https://github.com/apache/spark/pull/18594]

> Task failures during shutdown cause problems with preempted executors
> ---------------------------------------------------------------------
>
>                 Key: SPARK-20904
>                 URL: https://issues.apache.org/jira/browse/SPARK-20904
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core, YARN
>    Affects Versions: 1.6.0
>            Reporter: Marcelo Vanzin
>             Fix For: 2.2.1, 2.3.0
>
>
> Spark runs tasks in a thread pool that uses daemon threads in each executor. That means
that when the JVM gets a signal to shut down, those tasks keep running.
> Now when YARN preempts an executor, it sends a SIGTERM to the process, triggering the
JVM shutdown. That causes shutdown hooks to run which may cause user code running in those
tasks to fail, and report task failures to the driver. Those failures are then counted towards
the maximum number of allowed failures, even though in this case we don't want that because
the executor was preempted.
> So we need a better way to handle that situation.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message