spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From wenc...@apache.org
Subject spark git commit: [SPARK-22653] executorAddress registered in CoarseGrainedSchedulerBac…
Date Fri, 01 Dec 2017 02:53:28 GMT
Repository: spark
Updated Branches:
  refs/heads/master 7da1f5708 -> dc365422b


[SPARK-22653] executorAddress registered in CoarseGrainedSchedulerBac…

https://issues.apache.org/jira/browse/SPARK-22653
executorRef.address can be null, pass the executorAddress which accounts for it being null
a few lines above the fix.

Manually tested this patch. You can reproduce the issue by running a simple spark-shell in
yarn client mode with dynamic allocation and request some executors up front. Let those executors
idle timeout. Get a heap dump. Without this fix, you will see that addressToExecutorId still
contains the ids, with the fix addressToExecutorId is properly cleaned up.

Author: Thomas Graves <tgraves@oath.com>

Closes #19850 from tgravescs/SPARK-22653.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/dc365422
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/dc365422
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/dc365422

Branch: refs/heads/master
Commit: dc365422bb337d19ef39739c7c3cf9e53ec85d09
Parents: 7da1f57
Author: Thomas Graves <tgraves@oath.com>
Authored: Fri Dec 1 10:53:16 2017 +0800
Committer: Wenchen Fan <wenchen@databricks.com>
Committed: Fri Dec 1 10:53:16 2017 +0800

----------------------------------------------------------------------
 .../spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala    | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/dc365422/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
b/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
index 22d9c4c..7bfb4d5 100644
--- a/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
+++ b/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
@@ -182,7 +182,7 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val
rpcEnv: Rp
           addressToExecutorId(executorAddress) = executorId
           totalCoreCount.addAndGet(cores)
           totalRegisteredExecutors.addAndGet(1)
-          val data = new ExecutorData(executorRef, executorRef.address, hostname,
+          val data = new ExecutorData(executorRef, executorAddress, hostname,
             cores, cores, logUrls)
           // This must be synchronized because variables mutated
           // in this block are read when requesting executors


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message