spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Imran Rashid (JIRA)" <>
Subject [jira] [Resolved] (SPARK-24416) Update configuration definition for spark.blacklist.killBlacklistedExecutors
Date Tue, 12 Jun 2018 18:56:00 GMT


Imran Rashid resolved SPARK-24416.
       Resolution: Fixed
    Fix Version/s: 2.4.0

Issue resolved by pull request 21475

> Update configuration definition for spark.blacklist.killBlacklistedExecutors
> ----------------------------------------------------------------------------
>                 Key: SPARK-24416
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 2.3.0
>            Reporter: Sanket Reddy
>            Assignee: Sanket Reddy
>            Priority: Minor
>             Fix For: 2.4.0
> spark.blacklist.killBlacklistedExecutors is defined asĀ 
> (Experimental) If set to "true", allow Spark to automatically kill, and attempt to re-create,
executors when they are blacklisted. Note that, when an entire node is added to the blacklist,
all of the executors on that node will be killed.
> I presume the killing of blacklisted executors only happens after the stage completes
successfully and all tasks have completed or on fetch failures (updateBlacklistForFetchFailure/updateBlacklistForSuccessfulTaskSet).
It is confusing because the definition states that the executor will be attempted to be recreated
as soon as it is blacklisted. This is not true while the stage is in progress and an executor
is blacklisted, it will not attempt to cleanup until the stage finishes.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message