hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2369) Environment variable handling assumes values should be appended
Date Tue, 29 Jul 2014 13:59:38 GMT

    [ https://issues.apache.org/jira/browse/YARN-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14077729#comment-14077729
] 

Jason Lowe commented on YARN-2369:
----------------------------------

The code in question is in org.apache.hadoop.yarn.util.Apps#addToEnvironment:

{code}
  public static void addToEnvironment(
      Map<String, String> environment,
      String variable, String value, String classPathSeparator) {
    String val = environment.get(variable);
    if (val == null) {
      val = value;
    } else {
      val = val + classPathSeparator + value;
    }
    environment.put(StringInterner.weakIntern(variable), 
        StringInterner.weakIntern(val));
  }
{code}

This has very surprising results for any variable that isn't path-like.  For example, we ran
across a MapReduce job that had something like this in its environment settings:

yarn.app.mapreduce.am.env='JAVA_HOME=/inst/jdk,JAVA_HOME=/inst/jdk'

Rather than ending up with JAVA_HOME=/inst/jdk as one would expect, JAVA_HOME instead was
set to /inst/jdk:/inst/jdk which completely broke the job.

It seems to me that we should either use a whitelist of variables that support appending or
never append settings.  For the latter case if users desire values to be appended then they
can ask for it explicitly in their variable settings, like one of these forms depending upon
whether they want client-side environment variable expansion or container-side environment
variable expansion:
{noformat}
PATH='$PATH:/my/extra/path'
PATH='{{PATH}}:/my/extra/path'
{noformat}

> Environment variable handling assumes values should be appended
> ---------------------------------------------------------------
>
>                 Key: YARN-2369
>                 URL: https://issues.apache.org/jira/browse/YARN-2369
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 2.2.0
>            Reporter: Jason Lowe
>
> When processing environment variables for a container context the code assumes that the
value should be appended to any pre-existing value in the environment.  This may be desired
behavior for handling path-like environment variables such as PATH, LD_LIBRARY_PATH, CLASSPATH,
etc. but it is a non-intuitive and harmful way to handle any variable that does not have path-like
semantics.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message