spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-15067) YARN executors are launched with fixed perm gen size
Date Sun, 08 May 2016 09:25:12 GMT

    [ https://issues.apache.org/jira/browse/SPARK-15067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275527#comment-15275527
] 

Apache Spark commented on SPARK-15067:
--------------------------------------

User 'srowen' has created a pull request for this issue:
https://github.com/apache/spark/pull/12985

> YARN executors are launched with fixed perm gen size
> ----------------------------------------------------
>
>                 Key: SPARK-15067
>                 URL: https://issues.apache.org/jira/browse/SPARK-15067
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 1.6.0, 1.6.1
>            Reporter: Renato Falchi Brandão
>            Priority: Minor
>
> It is impossible to change the executors max perm gen size using the property "spark.executor.extraJavaOptions"
when you are running on YARN.
> When the JVM option "-XX:MaxPermSize" is set through the property "spark.executor.extraJavaOptions",
Spark put it properly in the shell command that will start the JVM container but, in the ending
of command, it sets again this option using a fixed value of 256m, as you can see in the log
I've extracted:
> 2016-04-30 17:20:12 INFO  ExecutorRunnable:58 -
> ===============================================================================
> YARN executor launch context:
>   env:
>     CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CONF_DIR<CPS>/usr/hdp/current/hadoop-client/*<CPS>/usr/hdp/current/hadoop-client/lib/*<CPS>/usr/hdp/current/hadoop-hdfs-client/*<CPS>/usr/hdp/current/hadoop-hdfs-client/lib/*<CPS>/usr/hdp/current/hadoop-yarn-client/*<CPS>/usr/hdp/current/hadoop-yarn-client/lib/*<CPS>/usr/hdp/mr-framework/hadoop/share/hadoop/mapreduce/*:/usr/hdp/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:/usr/hdp/mr-framework/hadoop/share/hadoop/common/*:/usr/hdp/mr-framework/hadoop/share/hadoop/common/lib/*:/usr/hdp/mr-framework/hadoop/share/hadoop/yarn/*:/usr/hdp/mr-framework/hadoop/share/hadoop/yarn/lib/*:/usr/hdp/mr-framework/hadoop/share/hadoop/hdfs/*:/usr/hdp/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/current/hadoop/lib/hadoop-lzo-0.6.0.jar:/etc/hadoop/conf/secure
>     SPARK_LOG_URL_STDERR -> http://xxxxx0668sl.xxxxx.br:8042/node/containerlogs/container_1456962126505_329993_01_000002/h_loadbd/stderr?start=-4096
>     SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1456962126505_329993
>     SPARK_YARN_CACHE_FILES_FILE_SIZES -> 191719054,166
>     SPARK_USER -> h_loadbd
>     SPARK_YARN_CACHE_FILES_VISIBILITIES -> PUBLIC,PUBLIC
>     SPARK_YARN_MODE -> true
>     SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1459806496093,1459808508343
>     SPARK_LOG_URL_STDOUT -> http://xxxxx0668sl.xxxxx.br:8042/node/containerlogs/container_1456962126505_329993_01_000002/h_loadbd/stdout?start=-4096
>     SPARK_YARN_CACHE_FILES -> hdfs://xxxxx/user/datalab/hdp/spark/lib/spark-assembly-1.6.0.2.3.4.1-10-hadoop2.7.1.2.3.4.1-10.jar#__spark__.jar,hdfs://tlvcluster/user/datalab/hdp/spark/conf/hive-site.xml#hive-site.xml
>   command:
>     {{JAVA_HOME}}/bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms6144m -Xmx6144m
'-XX:+PrintGCDetails' '-XX:MaxPermSize=1024M' '-XX:+PrintGCTimeStamps' -Djava.io.tmpdir={{PWD}}/tmp
'-Dspark.akka.timeout=300000' '-Dspark.driver.port=62875' '-Dspark.rpc.askTimeout=300000'
'-Dspark.rpc.lookupTimeout=300000' -Dspark.yarn.app.container.log.dir=<LOG_DIR> -XX:MaxPermSize=256m
org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@10.125.81.42:62875
--executor-id 1 --hostname xxxxx0668sl.xxxxx.br --cores 1 --app-id application_1456962126505_329993
--user-class-path file:$PWD/__app__.jar 1> <LOG_DIR>/stdout 2> <LOG_DIR>/stderr
> Analyzing the code is possible to see that all the options set in the property "spark.executor.extraJavaOptions"
are enclosed, one by one, in single quotes (ExecutorRunnable.scala:151) before the launcher
take the decision if a default value has to be provided or not for the option "-XX:MaxPermSize"
(ExecutorRunnable.scala:202).
> This decision is taken examining all the options set and looking for a string starting
with the value "-XX:MaxPermSize" (CommandBuilderUtils.java:328). If that value is not found,
the default value is set.
> A string option starting without single quote will never be found, then, a default value
will always be provided.
> A possible solution is change the source code of CommandBuilderUtils.java in the line
328:
> From-> if (arg.startsWith("-XX:MaxPermSize="))
> To-> if (arg.indexOf("-XX:MaxPermSize=") > -1)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message