Return-Path: X-Original-To: apmail-spark-commits-archive@minotaur.apache.org Delivered-To: apmail-spark-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 552A7109B4 for ; Sun, 16 Feb 2014 20:26:24 +0000 (UTC) Received: (qmail 7519 invoked by uid 500); 16 Feb 2014 20:26:23 -0000 Delivered-To: apmail-spark-commits-archive@spark.apache.org Received: (qmail 7489 invoked by uid 500); 16 Feb 2014 20:26:23 -0000 Mailing-List: contact commits-help@spark.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@spark.incubator.apache.org Delivered-To: mailing list commits@spark.incubator.apache.org Received: (qmail 7481 invoked by uid 99); 16 Feb 2014 20:26:23 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 16 Feb 2014 20:26:23 +0000 X-ASF-Spam-Status: No, hits=-2000.6 required=5.0 tests=ALL_TRUSTED,RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.3] (HELO mail.apache.org) (140.211.11.3) by apache.org (qpsmtpd/0.29) with SMTP; Sun, 16 Feb 2014 20:26:20 +0000 Received: (qmail 7077 invoked by uid 99); 16 Feb 2014 20:25:58 -0000 Received: from tyr.zones.apache.org (HELO tyr.zones.apache.org) (140.211.11.114) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 16 Feb 2014 20:25:58 +0000 Received: by tyr.zones.apache.org (Postfix, from userid 65534) id 11E0082776B; Sun, 16 Feb 2014 20:25:58 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: adav@apache.org To: commits@spark.incubator.apache.org Message-Id: <464c2e499912463db9745042dd5e3ea3@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: git commit: [SPARK-1092] print warning information if user use SPARK_MEM to regulate executor memory usage Date: Sun, 16 Feb 2014 20:25:57 +0000 (UTC) X-Virus-Checked: Checked by ClamAV on apache.org Repository: incubator-spark Updated Branches: refs/heads/master eec4bd1a1 -> 1cad38138 [SPARK-1092] print warning information if user use SPARK_MEM to regulate executor memory usage https://spark-project.atlassian.net/browse/SPARK-1092?jql=project%20%3D%20SPARK print warning information if user set SPARK_MEM to regulate memory usage of executors ---- OUTDATED: Currently, users will usually set SPARK_MEM to control the memory usage of driver programs, (in spark-class) 91 JAVA_OPTS="$OUR_JAVA_OPTS" 92 JAVA_OPTS="$JAVA_OPTS -Djava.library.path=$SPARK_LIBRARY_PATH" 93 JAVA_OPTS="$JAVA_OPTS -Xms$SPARK_MEM -Xmx$SPARK_MEM" if they didn't set spark.executor.memory, the value in this environment variable will also affect the memory usage of executors, because the following lines in SparkContext privatespark val executorMemory = conf.getOption("spark.executor.memory") .orElse(Option(System.getenv("SPARK_MEM"))) .map(Utils.memoryStringToMb) .getOrElse(512) also since SPARK_MEM has been (proposed to) deprecated in SPARK-929 (https://spark-project.atlassian.net/browse/SPARK-929) and the corresponding PR (https://github.com/apache/incubator-spark/pull/104) we should remove this line Author: CodingCat Closes #602 from CodingCat/clean_spark_mem and squashes the following commits: 302bb28 [CodingCat] print warning information if user use SPARK_MEM to regulate executor memory usage Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/1cad3813 Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/1cad3813 Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/1cad3813 Branch: refs/heads/master Commit: 1cad3813879cf6a968cfbf427da37fbb4f39dc86 Parents: eec4bd1 Author: CodingCat Authored: Sun Feb 16 12:25:38 2014 -0800 Committer: Aaron Davidson Committed: Sun Feb 16 12:25:38 2014 -0800 ---------------------------------------------------------------------- core/src/main/scala/org/apache/spark/SparkContext.scala | 5 +++++ 1 file changed, 5 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/1cad3813/core/src/main/scala/org/apache/spark/SparkContext.scala ---------------------------------------------------------------------- diff --git a/core/src/main/scala/org/apache/spark/SparkContext.scala b/core/src/main/scala/org/apache/spark/SparkContext.scala index 25f7a5e..5a6d06b 100644 --- a/core/src/main/scala/org/apache/spark/SparkContext.scala +++ b/core/src/main/scala/org/apache/spark/SparkContext.scala @@ -170,6 +170,11 @@ class SparkContext( .map(Utils.memoryStringToMb) .getOrElse(512) + if (!conf.contains("spark.executor.memory") && sys.env.contains("SPARK_MEM")) { + logWarning("Using SPARK_MEM to set amount of memory to use per executor process is " + + "deprecated, instead use spark.executor.memory") + } + // Environment variables to pass to our executors private[spark] val executorEnvs = HashMap[String, String]() // Note: SPARK_MEM is included for Mesos, but overwritten for standalone mode in ExecutorRunner