hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
Date Fri, 22 Sep 2017 13:04:02 GMT

    [ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16176376#comment-16176376

Jason Lowe commented on YARN-7226:

bq.  For example, an image used for running MR tasks could specify its own JAVA_HOME that
needs to be used instead of the JAVA_HOME that is specified in the nodemanager's environment.

But that's not what the whitelist is for.  The whitelist is for the *nodemanager's environment
variables* not the variable associated with the Docker image that the user requested to run
with their job.  That's not a nodemanager thing, that's a user's Docker image thing.  In all
cases, a whitelist should never prevent the container from overriding the variable, but that
appears to be what this is doing for the Docker case.  Any variable in the whitelist will
not be set to the user's value _unless_ the Docker image doesn't specify it, correct?  If
so, that's going to be surprising to users.

If Docker wants/needs the ability to "screen" certain variables from ever being set if they
are specified in the image, whether that setting comes from the container context directly
or the NM's environment via the whitelist, then IMHO that should be a separate config.  Otherwise
in a mixed-mode environment where some containers are running outside of Docker and some are
running in Docker, how is an admin supposed to properly set yarn.nodemanager.env-whitelist
properly?  The set of variables in a Docker image and the set of NM variables that need to
peek through to non-Docker containers may not be the same.

> Whitelisted variables do not support delayed variable expansion
> ---------------------------------------------------------------
>                 Key: YARN-7226
>                 URL: https://issues.apache.org/jira/browse/YARN-7226
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>         Attachments: YARN-7226.001.patch, YARN-7226.002.patch
> The nodemanager supports a configurable list of environment variables, via yarn.nodemanager.env-whitelist,
that will be propagated to the container's environment unless those variables were specified
in the container launch context.  Unfortunately the handling of these whitelisted variables
prevents using delayed variable expansion.  For example, if a user shipped their own version
of hadoop with their job via the distributed cache and specified:
> {noformat}
> HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/
> {noformat}
>  as part of their job, the variable will be set as the *literal* string:
> {noformat}
> $PWD/my-private-hadoop/
> {noformat}
> rather than having $PWD expand to the container's current directory as it does for any
other, non-whitelisted variable being set to the same value.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message