hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lingfeng Su (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-7062) yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
Date Sun, 20 Aug 2017 18:08:00 GMT

     [ https://issues.apache.org/jira/browse/YARN-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Lingfeng Su updated YARN-7062:
------------------------------
    Description: 
I passed a json string like "{User: Billy, {Age: 18}}" to main method args, when running spark
jobs on yarn.

It was changed by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: 18" ("}}" to
"")

I found the final arg in launch_container.sh of yarn containers, as:
 --arg '{User: Billy, {Age: 18'

{code:java}
exec /bin/bash -c "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH"
$JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001
org.apache.spark.deploy.yarn.ApplicationMaster --class 'org.apache.spark.examples.sql.hive.HiveFromSpark'
--jar file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' --properties-file
$PWD/__spark_conf__/__spark_conf__.properties 1> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001/stdout
2> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001/stderr"

{code}

We could make some improvements

method:
{code:java}
@VisibleForTesting
  public static String expandEnvironment(String var,
      Path containerLogDir) {
    var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR,
      containerLogDir.toString());
    var =  var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR,
      File.pathSeparator);

    // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced
    // as %VAR% and on Linux replaced as "$VAR"
    if (Shell.WINDOWS) {
      var = var.replaceAll("(\\{\\{)|(\\}\\})", "%");
    } else {
      var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$");
      var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, "");
    }
    return var;
  }
{code}




  was:
I passed a json string like "{User: Billy, {Age: 18}}" to main method args, when running spark
jobs on yarn.

It was changed by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: 18" ("}}" to
"")

I found the final arg in launch_container.sh of yarn containers, as:
 --arg '{User: Billy, {Age: 18'

{code:java}
exec /bin/bash -c "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH"
$JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001
org.apache.spark.deploy.yarn.ApplicationMaster --class 'org.apache.spark.examples.sql.hive.HiveFromSpark'
--jar file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' --properties-file
$PWD/__spark_conf__/__spark_conf__.properties 1> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001/stdout
2> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001/stderr"

{code}

We could make some improvements




> yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
> -----------------------------------------------------------------------------------
>
>                 Key: YARN-7062
>                 URL: https://issues.apache.org/jira/browse/YARN-7062
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: applications
>            Reporter: Lingfeng Su
>            Assignee: Lingfeng Su
>
> I passed a json string like "{User: Billy, {Age: 18}}" to main method args, when running
spark jobs on yarn.
> It was changed by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: 18" ("}}"
to "")
> I found the final arg in launch_container.sh of yarn containers, as:
>  --arg '{User: Billy, {Age: 18'
> {code:java}
> exec /bin/bash -c "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH"
$JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001
org.apache.spark.deploy.yarn.ApplicationMaster --class 'org.apache.spark.examples.sql.hive.HiveFromSpark'
--jar file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' --properties-file
$PWD/__spark_conf__/__spark_conf__.properties 1> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001/stdout
2> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001/stderr"
> {code}
> We could make some improvements
> method:
> {code:java}
> @VisibleForTesting
>   public static String expandEnvironment(String var,
>       Path containerLogDir) {
>     var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR,
>       containerLogDir.toString());
>     var =  var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR,
>       File.pathSeparator);
>     // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced
>     // as %VAR% and on Linux replaced as "$VAR"
>     if (Shell.WINDOWS) {
>       var = var.replaceAll("(\\{\\{)|(\\}\\})", "%");
>     } else {
>       var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$");
>       var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, "");
>     }
>     return var;
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message