hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lingfeng Su (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-7062) yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
Date Sun, 20 Aug 2017 18:06:09 GMT

     [ https://issues.apache.org/jira/browse/YARN-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Lingfeng Su updated YARN-7062:
------------------------------
    Description: 
I passed a json string like "{User: Billy, {Age: 18}}" to main method args, when running spark
jobs on yarn.

It may exchange by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: 18" ("}}" to
"")

I found the final arg in launch_container.sh of yarn containers, as:
 --arg '{User: Billy, {Age: 18'

{code:java}
exec /bin/bash -c "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH"
$JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001
org.apache.spark.deploy.yarn.ApplicationMaster --class 'org.apache.spark.examples.sql.hive.HiveFromSpark'
--jar file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' --properties-file
$PWD/__spark_conf__/__spark_conf__.properties 1> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001/stdout
2> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001/stderr"

{code}

We could make some improvements



  was:
I passed a json string like "{User: Billy, {Age: 18}}" to main method args, when running spark
jobs on yarn.
It may exchange by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: 18" ("}}" to
"")

I found the final arg in launch_container.sh of yarn containers, as:
 --arg '{User: Billy, {Age: 18'

{code:java}
exec /bin/bash -c "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH"
$JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001
org.apache.spark.deploy.yarn.ApplicationMaster --class 'org.apache.spark.examples.sql.hive.HiveFromSpark'
--jar file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' --properties-file
$PWD/__spark_conf__/__spark_conf__.properties 1> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001/stdout
2> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001/stderr"

{code}

We could make some improvements




> yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
> -----------------------------------------------------------------------------------
>
>                 Key: YARN-7062
>                 URL: https://issues.apache.org/jira/browse/YARN-7062
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: applications
>            Reporter: Lingfeng Su
>            Assignee: Lingfeng Su
>
> I passed a json string like "{User: Billy, {Age: 18}}" to main method args, when running
spark jobs on yarn.
> It may exchange by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: 18" ("}}"
to "")
> I found the final arg in launch_container.sh of yarn containers, as:
>  --arg '{User: Billy, {Age: 18'
> {code:java}
> exec /bin/bash -c "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH"
$JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001
org.apache.spark.deploy.yarn.ApplicationMaster --class 'org.apache.spark.examples.sql.hive.HiveFromSpark'
--jar file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' --properties-file
$PWD/__spark_conf__/__spark_conf__.properties 1> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001/stdout
2> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001/stderr"
> {code}
> We could make some improvements



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message