spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sro...@apache.org
Subject spark git commit: [SPARK-5157][YARN] Configure more JVM options properly when we use ConcMarkSweepGC for AM.
Date Fri, 06 Feb 2015 09:39:19 GMT
Repository: spark
Updated Branches:
  refs/heads/master f6ba813af -> 24dbc50b9


[SPARK-5157][YARN] Configure more JVM options properly when we use ConcMarkSweepGC for AM.

When we set `SPARK_USE_CONC_INCR_GC`, ConcurrentMarkSweepGC works on the AM.
Actually, if ConcurrentMarkSweepGC is set for the JVM, following JVM options are set automatically
and implicitly.

* MaxTenuringThreshold=0
* SurvivorRatio=1024

Those can not be proper value for most cases.
See also http://www.oracle.com/technetwork/java/tuning-139912.html

Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp>

Closes #3956 from sarutak/SPARK-5157 and squashes the following commits:

c15da4e [Kousuke Saruta] Set more JVM options for AM when enabling CMS


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/24dbc50b
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/24dbc50b
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/24dbc50b

Branch: refs/heads/master
Commit: 24dbc50b934bfcd9fc3fd91cfd9966b75b250ad7
Parents: f6ba813
Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp>
Authored: Fri Feb 6 09:39:12 2015 +0000
Committer: Sean Owen <sowen@cloudera.com>
Committed: Fri Feb 6 09:39:12 2015 +0000

----------------------------------------------------------------------
 yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala | 2 ++
 1 file changed, 2 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/24dbc50b/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
----------------------------------------------------------------------
diff --git a/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala b/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
index 91e8574..e700509 100644
--- a/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
+++ b/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
@@ -418,6 +418,8 @@ private[spark] class Client(
       // In our expts, using (default) throughput collector has severe perf ramifications
in
       // multi-tenant machines
       javaOpts += "-XX:+UseConcMarkSweepGC"
+      javaOpts += "-XX:MaxTenuringThreshold=31"
+      javaOpts += "-XX:SurvivorRatio=8"
       javaOpts += "-XX:+CMSIncrementalMode"
       javaOpts += "-XX:+CMSIncrementalPacing"
       javaOpts += "-XX:CMSIncrementalDutyCycleMin=0"


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message