spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From iras...@apache.org
Subject spark git commit: [SPARK-24416] Fix configuration specification for killBlacklisted executors
Date Tue, 12 Jun 2018 18:55:14 GMT
Repository: spark
Updated Branches:
  refs/heads/master 2824f1436 -> 3af1d3e6d


[SPARK-24416] Fix configuration specification for killBlacklisted executors

## What changes were proposed in this pull request?

spark.blacklist.killBlacklistedExecutors is defined as

(Experimental) If set to "true", allow Spark to automatically kill, and attempt to re-create,
executors when they are blacklisted. Note that, when an entire node is added to the blacklist,
all of the executors on that node will be killed.

I presume the killing of blacklisted executors only happens after the stage completes successfully
and all tasks have completed or on fetch failures (updateBlacklistForFetchFailure/updateBlacklistForSuccessfulTaskSet).
It is confusing because the definition states that the executor will be attempted to be recreated
as soon as it is blacklisted. This is not true while the stage is in progress and an executor
is blacklisted, it will not attempt to cleanup until the stage finishes.

Author: Sanket Chintapalli <schintap@yahoo-inc.com>

Closes #21475 from redsanket/SPARK-24416.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/3af1d3e6
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/3af1d3e6
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/3af1d3e6

Branch: refs/heads/master
Commit: 3af1d3e6d95719e15a997877d5ecd3bb40c08b9c
Parents: 2824f14
Author: Sanket Chintapalli <schintap@yahoo-inc.com>
Authored: Tue Jun 12 13:55:08 2018 -0500
Committer: Imran Rashid <irashid@cloudera.com>
Committed: Tue Jun 12 13:55:08 2018 -0500

----------------------------------------------------------------------
 docs/configuration.md | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/3af1d3e6/docs/configuration.md
----------------------------------------------------------------------
diff --git a/docs/configuration.md b/docs/configuration.md
index 5588c37..6aa7878 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -1656,9 +1656,10 @@ Apart from these, the following properties are also available, and
may be useful
   <td><code>spark.blacklist.killBlacklistedExecutors</code></td>
   <td>false</td>
   <td>
-    (Experimental) If set to "true", allow Spark to automatically kill, and attempt to re-create,
-    executors when they are blacklisted.  Note that, when an entire node is added to the
blacklist,
-    all of the executors on that node will be killed.
+    (Experimental) If set to "true", allow Spark to automatically kill the executors 
+    when they are blacklisted on fetch failure or blacklisted for the entire application,

+    as controlled by spark.blacklist.application.*. Note that, when an entire node is added

+    to the blacklist, all of the executors on that node will be killed.
   </td>
 </tr>
 <tr>


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message