spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject spark git commit: [SPARK-22653] executorAddress registered in CoarseGrainedSchedulerBac…
Date Fri, 01 Dec 2017 02:53:28 GMT
Repository: spark
Updated Branches:
  refs/heads/master 7da1f5708 -> dc365422b

[SPARK-22653] executorAddress registered in CoarseGrainedSchedulerBac…
executorRef.address can be null, pass the executorAddress which accounts for it being null
a few lines above the fix.

Manually tested this patch. You can reproduce the issue by running a simple spark-shell in
yarn client mode with dynamic allocation and request some executors up front. Let those executors
idle timeout. Get a heap dump. Without this fix, you will see that addressToExecutorId still
contains the ids, with the fix addressToExecutorId is properly cleaned up.

Author: Thomas Graves <>

Closes #19850 from tgravescs/SPARK-22653.


Branch: refs/heads/master
Commit: dc365422bb337d19ef39739c7c3cf9e53ec85d09
Parents: 7da1f57
Author: Thomas Graves <>
Authored: Fri Dec 1 10:53:16 2017 +0800
Committer: Wenchen Fan <>
Committed: Fri Dec 1 10:53:16 2017 +0800

 .../spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala    | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
index 22d9c4c..7bfb4d5 100644
--- a/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
+++ b/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
@@ -182,7 +182,7 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val
rpcEnv: Rp
           addressToExecutorId(executorAddress) = executorId
-          val data = new ExecutorData(executorRef, executorRef.address, hostname,
+          val data = new ExecutorData(executorRef, executorAddress, hostname,
             cores, cores, logUrls)
           // This must be synchronized because variables mutated
           // in this block are read when requesting executors

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message