spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From andrewo...@apache.org
Subject git commit: [SPARK-3653] Respect SPARK_*_MEMORY for cluster mode
Date Tue, 23 Sep 2014 21:00:40 GMT
Repository: spark
Updated Branches:
  refs/heads/master d79238d03 -> b3fef50e2


[SPARK-3653] Respect SPARK_*_MEMORY for cluster mode

`SPARK_DRIVER_MEMORY` was only used to start the `SparkSubmit` JVM, which becomes the driver
only in client mode but not cluster mode. In cluster mode, this property is simply not propagated
to the worker nodes.

`SPARK_EXECUTOR_MEMORY` is picked up from `SparkContext`, but in cluster mode the driver runs
on one of the worker machines, where this environment variable may not be set.

Author: Andrew Or <andrewor14@gmail.com>

Closes #2500 from andrewor14/memory-env-vars and squashes the following commits:

6217b38 [Andrew Or] Respect SPARK_*_MEMORY for cluster mode


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b3fef50e
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b3fef50e
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b3fef50e

Branch: refs/heads/master
Commit: b3fef50e22fb3fe499f627179d17836a92dcb33a
Parents: d79238d
Author: Andrew Or <andrewor14@gmail.com>
Authored: Tue Sep 23 14:00:33 2014 -0700
Committer: Andrew Or <andrewor14@gmail.com>
Committed: Tue Sep 23 14:00:33 2014 -0700

----------------------------------------------------------------------
 .../scala/org/apache/spark/deploy/SparkSubmitArguments.scala     | 4 ++++
 1 file changed, 4 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/b3fef50e/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala b/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala
index 92e0917..2b72c61 100644
--- a/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala
@@ -75,6 +75,10 @@ private[spark] class SparkSubmitArguments(args: Seq[String]) {
     defaultProperties
   }
 
+  // Respect SPARK_*_MEMORY for cluster mode
+  driverMemory = sys.env.get("SPARK_DRIVER_MEMORY").orNull
+  executorMemory = sys.env.get("SPARK_EXECUTOR_MEMORY").orNull
+
   parseOpts(args.toList)
   mergeSparkProperties()
   checkRequiredArguments()


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message