Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Sun, 20 Aug 2017 18:06:09 +0000 (UTC)
From: "Lingfeng Su (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.13096307.1503252084000.74749.1503252369952@Atlassian.JIRA>
In-Reply-To: <JIRA.13096307.1503252084000@Atlassian.JIRA>
References: <JIRA.13096307.1503252084000@Atlassian.JIRA> <JIRA.13096307.1503252084981@jira-lw-us.apache.org>
Subject: [jira] [Updated] (YARN-7062) yarn job args changed by
 ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Sun, 20 Aug 2017 18:06:18 -0000


     [ https://issues.apache.org/jira/browse/YARN-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Lingfeng Su updated YARN-7062:
------------------------------
    Description: 
I passed a json string like "{User: Billy, {Age: 18}}" to main method args, when running spark jobs on yarn.

It may exchange by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: 18" ("}}" to "")

I found the final arg in launch_container.sh of yarn containers, as:
 --arg '{User: Billy, {Age: 18'

{code:java}
exec /bin/bash -c "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001 org.apache.spark.deploy.yarn.ApplicationMaster --class 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001/stdout 2> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001/stderr"

{code}

We could make some improvements


  was:
I passed a json string like "{User: Billy, {Age: 18}}" to main method args, when running spark jobs on yarn.
It may exchange by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: 18" ("}}" to "")

I found the final arg in launch_container.sh of yarn containers, as:
 --arg '{User: Billy, {Age: 18'

{code:java}
exec /bin/bash -c "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001 org.apache.spark.deploy.yarn.ApplicationMaster --class 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001/stdout 2> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001/stderr"

{code}

We could make some improvements


> yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
> -----------------------------------------------------------------------------------
>
>                 Key: YARN-7062
>                 URL: https://issues.apache.org/jira/browse/YARN-7062
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: applications
>            Reporter: Lingfeng Su
>            Assignee: Lingfeng Su
>
> I passed a json string like "{User: Billy, {Age: 18}}" to main method args, when running spark jobs on yarn.
> It may exchange by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: 18" ("}}" to "")
> I found the final arg in launch_container.sh of yarn containers, as:
>  --arg '{User: Billy, {Age: 18'
> {code:java}
> exec /bin/bash -c "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001 org.apache.spark.deploy.yarn.ApplicationMaster --class 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001/stdout 2> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001/stderr"
> {code}
> We could make some improvements


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org