spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From van...@apache.org
Subject spark git commit: [SPARK-12142][CORE]Reply false when container allocator is not ready and reset target
Date Sat, 05 Dec 2015 00:51:08 GMT
Repository: spark
Updated Branches:
  refs/heads/master b7204e1d4 -> bbfc16ec9


[SPARK-12142][CORE]Reply false when container allocator is not ready and reset target

Using Dynamic Allocation function, when a new AM is starting, and ExecutorAllocationManager
send RequestExecutor message to AM. If the container allocator is not ready, the whole app
will hang on

Author: meiyoula <1039320815@qq.com>

Closes #10138 from XuTingjun/patch-1.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/bbfc16ec
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/bbfc16ec
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/bbfc16ec

Branch: refs/heads/master
Commit: bbfc16ec9d690c2dfa20896bd6d33f9783b9c109
Parents: b7204e1
Author: meiyoula <1039320815@qq.com>
Authored: Fri Dec 4 16:50:40 2015 -0800
Committer: Marcelo Vanzin <vanzin@cloudera.com>
Committed: Fri Dec 4 16:50:43 2015 -0800

----------------------------------------------------------------------
 .../main/scala/org/apache/spark/ExecutorAllocationManager.scala   | 1 +
 .../scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala    | 3 ++-
 2 files changed, 3 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/bbfc16ec/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala b/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
index 6419218..34c32ce 100644
--- a/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
+++ b/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
@@ -370,6 +370,7 @@ private[spark] class ExecutorAllocationManager(
     } else {
       logWarning(
         s"Unable to reach the cluster manager to request $numExecutorsTarget total executors!")
+      numExecutorsTarget = oldNumExecutorsTarget
       0
     }
   }

http://git-wip-us.apache.org/repos/asf/spark/blob/bbfc16ec/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
----------------------------------------------------------------------
diff --git a/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala b/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
index 13ef4df..1970f7d 100644
--- a/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
+++ b/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
@@ -600,11 +600,12 @@ private[spark] class ApplicationMaster(
               localityAwareTasks, hostToLocalTaskCount)) {
               resetAllocatorInterval()
             }
+            context.reply(true)
 
           case None =>
             logWarning("Container allocator is not ready to request executors yet.")
+            context.reply(false)
         }
-        context.reply(true)
 
       case KillExecutors(executorIds) =>
         logInfo(s"Driver requested to kill executor(s) ${executorIds.mkString(", ")}.")


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message