hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lingfeng Su (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-7062) yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
Date Tue, 22 Aug 2017 16:21:02 GMT

     [ https://issues.apache.org/jira/browse/YARN-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Lingfeng Su updated YARN-7062:
------------------------------
    Attachment: YARN-7062.001.patch

> yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
> -----------------------------------------------------------------------------------
>
>                 Key: YARN-7062
>                 URL: https://issues.apache.org/jira/browse/YARN-7062
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: applications
>            Reporter: Lingfeng Su
>            Assignee: Lingfeng Su
>              Labels: patch
>         Attachments: YARN-7062.001.patch
>
>
> I passed a json string like "{User: Billy, {Age: 18}}" to main method args, when running
spark jobs on yarn.
> It was changed by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: 18" ("}}"
to "")
> I found the final arg in launch_container.sh of yarn containers, as:
>  --arg '{User: Billy, {Age: 18'
> {code:java}
> exec /bin/bash -c "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH"
$JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001
org.apache.spark.deploy.yarn.ApplicationMaster --class 'org.apache.spark.examples.sql.hive.HiveFromSpark'
--jar file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' --properties-file
$PWD/__spark_conf__/__spark_conf__.properties 1> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001/stdout
2> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001/stderr"
> {code}
> We could make some improvements
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.expandEnvironment():

> {code:java}
> @VisibleForTesting
>   public static String expandEnvironment(String var,
>       Path containerLogDir) {
>     var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR,
>       containerLogDir.toString());
>     var =  var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR,
>       File.pathSeparator);
>     // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced
>     // as %VAR% and on Linux replaced as "$VAR"
>     if (Shell.WINDOWS) {
>       var = var.replaceAll("(\\{\\{)|(\\}\\})", "%");
>     } else {
>       var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$");
>       var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, "");
>     }
>     return var;
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message