spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From r...@apache.org
Subject git commit: Merge pull request #526 from tgravescs/yarn_client_stop_am_fix. Closes #526.
Date Thu, 06 Feb 2014 07:37:42 GMT
Updated Branches:
  refs/heads/branch-0.9 d815cfa68 -> b044b0b4f


Merge pull request #526 from tgravescs/yarn_client_stop_am_fix. Closes #526.

spark on yarn - yarn-client mode doesn't always exit immediately

https://spark-project.atlassian.net/browse/SPARK-1049

If you run in the yarn-client mode but you don't get all the workers you requested right away
and then you exit your application, the application master stays around until it gets the
number of workers you initially requested. This is a waste of resources.  The AM should exit
immediately upon the client going away.

This fix simply checks to see if the driver closed while its waiting for the initial # of
workers.

Author: Thomas Graves <tgraves@apache.org>

== Merge branch commits ==

commit 03f40a62584b6bdd094ba91670cd4aa6afe7cd81
Author: Thomas Graves <tgraves@apache.org>
Date:   Fri Jan 31 11:23:10 2014 -0600

    spark on yarn - yarn-client mode doesn't always exit immediately

(cherry picked from commit 38020961d101e792393855fd00d8e42f40713754)
Signed-off-by: Reynold Xin <rxin@apache.org>


Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/b044b0b4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/b044b0b4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/b044b0b4

Branch: refs/heads/branch-0.9
Commit: b044b0b4fa07ec92fd2c80589247d59a5925df96
Parents: d815cfa
Author: Thomas Graves <tgraves@apache.org>
Authored: Wed Feb 5 23:37:07 2014 -0800
Committer: Reynold Xin <rxin@apache.org>
Committed: Wed Feb 5 23:37:38 2014 -0800

----------------------------------------------------------------------
 .../main/scala/org/apache/spark/deploy/yarn/WorkerLauncher.scala   | 2 +-
 .../main/scala/org/apache/spark/deploy/yarn/WorkerLauncher.scala   | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/b044b0b4/yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/WorkerLauncher.scala
----------------------------------------------------------------------
diff --git a/yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/WorkerLauncher.scala b/yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/WorkerLauncher.scala
index 9fe4d64..138c279 100644
--- a/yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/WorkerLauncher.scala
+++ b/yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/WorkerLauncher.scala
@@ -210,7 +210,7 @@ class WorkerLauncher(args: ApplicationMasterArguments, conf: Configuration,
spar
     // Wait until all containers have finished
     // TODO: This is a bit ugly. Can we make it nicer?
     // TODO: Handle container failure
-    while(yarnAllocator.getNumWorkersRunning < args.numWorkers) {
+    while ((yarnAllocator.getNumWorkersRunning < args.numWorkers) && (!driverClosed))
{
       yarnAllocator.allocateContainers(math.max(args.numWorkers - yarnAllocator.getNumWorkersRunning,
0))
       Thread.sleep(100)
     }

http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/b044b0b4/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/WorkerLauncher.scala
----------------------------------------------------------------------
diff --git a/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/WorkerLauncher.scala
b/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/WorkerLauncher.scala
index 7835322..40600f3 100644
--- a/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/WorkerLauncher.scala
+++ b/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/WorkerLauncher.scala
@@ -193,7 +193,7 @@ class WorkerLauncher(args: ApplicationMasterArguments, conf: Configuration,
spar
     // TODO: Handle container failure
 
     yarnAllocator.addResourceRequests(args.numWorkers)
-    while (yarnAllocator.getNumWorkersRunning < args.numWorkers) {
+    while ((yarnAllocator.getNumWorkersRunning < args.numWorkers) && (!driverClosed))
{
       yarnAllocator.allocateResources()
       Thread.sleep(100)
     }


Mime
View raw message