hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sidharta Seethana (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
Date Thu, 21 Sep 2017 23:22:02 GMT

    [ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175635#comment-16175635
] 

Sidharta Seethana commented on YARN-7226:
-----------------------------------------

{quote}
As I understand it, a whitelisted variable is simply an environment variable that will be
propagated from the nodemanager's environment to the container's environment unless the container
has specified it's own value for that variable.
{quote}

[~jlowe],
Using docker containers introduces a new way in which environment variables for a container
can be specified - this is through the docker image itself, which is a fairly common scenario.
For example, an image used for running MR tasks could specify its own JAVA_HOME that needs
to be used instead of the JAVA_HOME that is specified in the nodemanager's environment. Since
{{launch_container.sh}} runs inside the docker container, using the specified docker image,
using {{putEnvIfAbsent}} for whitelisted env vars doesn't do the right thing in this case.


> Whitelisted variables do not support delayed variable expansion
> ---------------------------------------------------------------
>
>                 Key: YARN-7226
>                 URL: https://issues.apache.org/jira/browse/YARN-7226
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>         Attachments: YARN-7226.001.patch, YARN-7226.002.patch
>
>
> The nodemanager supports a configurable list of environment variables, via yarn.nodemanager.env-whitelist,
that will be propagated to the container's environment unless those variables were specified
in the container launch context.  Unfortunately the handling of these whitelisted variables
prevents using delayed variable expansion.  For example, if a user shipped their own version
of hadoop with their job via the distributed cache and specified:
> {noformat}
> HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/
> {noformat}
>  as part of their job, the variable will be set as the *literal* string:
> {noformat}
> $PWD/my-private-hadoop/
> {noformat}
> rather than having $PWD expand to the container's current directory as it does for any
other, non-whitelisted variable being set to the same value.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message