hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "balloons (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-7277) Container Launch expand environment needs to consider bracket matching
Date Sat, 30 Sep 2017 04:08:00 GMT

     [ https://issues.apache.org/jira/browse/YARN-7277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

balloons updated YARN-7277:
---------------------------
    Description: 
The SPARK application I submitted always failed and I finally found that the commands I specified
to launch AM Container were changed by NM.

*The following is part of the excerpt I submitted to RM to see the command:*
{code:java}
*'{\"handler\":\"FILLER\",\"inputTable\":\"engine_arch.adult_train\",\"outputTable\":[\"ether_features_filler_\$experimentId_\$taskId_out0\"],\"params\":{\"age\":{\"param\":[\"0\"]}}}'*
{code}

*The following is an excerpt from the corresponding command used when I observe the NM launch
container:*
{code:java}
*'{\"handler\":\"FILLER\",\"inputTable\":\"engine_arch.adult_train\",\"outputTable\":[\"ether_features_filler_\$experimentId_\$taskId_out0\"],\"params\":{\"age\":{\"param\":[\"0\"]}*
{code}

Finally, I found that NM made the following transformation in launch container which led to
this situation:
{code:java}
@VisibleForTesting
  public static String expandEnvironment(String var,
      Path containerLogDir) {
    var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR,
      containerLogDir.toString());
    var =  var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR,
      File.pathSeparator);

    // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced
    // as %VAR% and on Linux replaced as "$VAR"
    if (Shell.WINDOWS) {
      var = var.replaceAll("(\\{\\{)|(\\}\\})", "%");
    } else {
      var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$");
      *var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, "");*
    }
    return var;
  }
{code}

I think this is a Bug that doesn't even consider the pairing of "*{{*" and "*}}*" when substituting,
but simply substituting for simple violence.


  was:
The SPARK application I submitted always failed and I finally found that the commands I specified
to launch AM Container were changed by NM.

*The following is part of the excerpt I submitted to RM to see the command:*
{code:java}
*'{\"handler\":\"FILLER\",\"inputTable\":\"engine_arch.adult_train\",\"outputTable\":[\"ether_features_filler_\$experimentId_\$taskId_out0\"],\"params\":{\"age\":{\"param\":[\"0\"]}}}'*
{code}

*The following is an excerpt from the corresponding command used when I observe the NM launch
container:*
{code:java}
*'{\"handler\":\"FILLER\",\"inputTable\":\"engine_arch.adult_train\",\"outputTable\":[\"ether_features_filler_\$experimentId_\$taskId_out0\"],\"params\":{\"age\":{\"param\":[\"0\"]}*
{code}

Finally, I found that NM made the following transformation in launch container which led to
this situation:
{code:java}
@VisibleForTesting
  public static String expandEnvironment(String var,
      Path containerLogDir) {
    var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR,
      containerLogDir.toString());
    var =  var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR,
      File.pathSeparator);

    // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced
    // as %VAR% and on Linux replaced as "$VAR"
    if (Shell.WINDOWS) {
      var = var.replaceAll("(\\{\\{)|(\\}\\})", "%");
    } else {
      var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$");
*{color:#f6c342}      var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, "");{color}*
    }
    return var;
  }
{code}

I think this is a Bug that doesn't even consider the pairing of "{{" and "}}" when substituting,
but simply substituting for simple violence.



> Container Launch expand environment needs to consider bracket matching
> ----------------------------------------------------------------------
>
>                 Key: YARN-7277
>                 URL: https://issues.apache.org/jira/browse/YARN-7277
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>            Reporter: balloons
>            Priority: Critical
>
> The SPARK application I submitted always failed and I finally found that the commands
I specified to launch AM Container were changed by NM.
> *The following is part of the excerpt I submitted to RM to see the command:*
> {code:java}
> *'{\"handler\":\"FILLER\",\"inputTable\":\"engine_arch.adult_train\",\"outputTable\":[\"ether_features_filler_\$experimentId_\$taskId_out0\"],\"params\":{\"age\":{\"param\":[\"0\"]}}}'*
> {code}
> *The following is an excerpt from the corresponding command used when I observe the NM
launch container:*
> {code:java}
> *'{\"handler\":\"FILLER\",\"inputTable\":\"engine_arch.adult_train\",\"outputTable\":[\"ether_features_filler_\$experimentId_\$taskId_out0\"],\"params\":{\"age\":{\"param\":[\"0\"]}*
> {code}
> Finally, I found that NM made the following transformation in launch container which
led to this situation:
> {code:java}
> @VisibleForTesting
>   public static String expandEnvironment(String var,
>       Path containerLogDir) {
>     var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR,
>       containerLogDir.toString());
>     var =  var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR,
>       File.pathSeparator);
>     // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced
>     // as %VAR% and on Linux replaced as "$VAR"
>     if (Shell.WINDOWS) {
>       var = var.replaceAll("(\\{\\{)|(\\}\\})", "%");
>     } else {
>       var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$");
>       *var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, "");*
>     }
>     return var;
>   }
> {code}
> I think this is a Bug that doesn't even consider the pairing of "*{{*" and "*}}*" when
substituting, but simply substituting for simple violence.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message