spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sro...@apache.org
Subject spark git commit: [SPARK-10332] [CORE] Fix yarn spark executor validation
Date Thu, 03 Sep 2015 08:31:15 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-1.5 94404ee53 -> f01a96713


[SPARK-10332] [CORE] Fix yarn spark executor validation

>From Jira:
Running spark-submit with yarn with number-executors equal to 0 when not using dynamic allocation
should error out.
In spark 1.5.0 it continues and ends up hanging.
yarn.ClientArguments still has the check so something else must have changed.
spark-submit --master yarn --deploy-mode cluster --class org.apache.spark.examples.SparkPi
--num-executors 0 ....
spark 1.4.1 errors with:
java.lang.IllegalArgumentException:
Number of executors was 0, but must be at least 1
(or 0 if dynamic executor allocation is enabled).

Author: Holden Karau <holden@pigscanfly.ca>

Closes #8580 from holdenk/SPARK-10332-spark-submit-to-yarn-executors-0-message.

(cherry picked from commit 67580f1f574d272af3712fd91458f3c87368c2e4)
Signed-off-by: Sean Owen <sowen@cloudera.com>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/f01a9671
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/f01a9671
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/f01a9671

Branch: refs/heads/branch-1.5
Commit: f01a96713a6ebb580c83e88652bc6d361aaec6f4
Parents: 94404ee
Author: Holden Karau <holden@pigscanfly.ca>
Authored: Thu Sep 3 09:30:54 2015 +0100
Committer: Sean Owen <sowen@cloudera.com>
Committed: Thu Sep 3 09:31:09 2015 +0100

----------------------------------------------------------------------
 .../main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala | 3 +++
 1 file changed, 3 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/f01a9671/yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala
----------------------------------------------------------------------
diff --git a/yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala b/yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala
index 4f42ffe..54f62e6 100644
--- a/yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala
+++ b/yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala
@@ -96,6 +96,9 @@ private[spark] class ClientArguments(args: Array[String], sparkConf: SparkConf)
       }
 
       numExecutors = initialNumExecutors
+    } else {
+      val numExecutorsConf = "spark.executor.instances"
+      numExecutors = sparkConf.getInt(numExecutorsConf, numExecutors)
     }
     principal = Option(principal)
       .orElse(sparkConf.getOption("spark.yarn.principal"))


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message