hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varun Saxena (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-5877) Allow all nm-whitelist-env to get overridden during launch
Date Sat, 19 Nov 2016 11:55:58 GMT

    [ https://issues.apache.org/jira/browse/YARN-5877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15679132#comment-15679132
] 

Varun Saxena edited comment on YARN-5877 at 11/19/16 11:55 AM:
---------------------------------------------------------------

Is this the right thing to do ?

NM whitelist configuration is used to allow containers to override NM's default environment.
Let us assume there is an environment variable DUMMY_ENV. Assume in some deployment, for some
jobs, its set in the ContainerLaunchContext and for all other jobs we want a default which
can be set in NM environment.
Before the change in the patch, we could have set DUMMY_ENV in NM whitelist and NM would have
picked up value for this environment variable from container launch context (if available)
otherwise taken it from NM environment.
After the change in this patch, NM would keep the value coming from container launch context
as default and use that only if the environment variable on NM is not set.

This is a departure from current behavior. Probably you can argue none of us are using it
this way but its still a change in behavior.
We have done the same for HADOOP_YARN_HOME, HADOOP_COMMON_HOME, etc. in YARN-3853 but these
are well known environment variables so probably nobody uses them the way I mentioned above.
Atleast we do not. So it may be safe to use them this way. 

If we were to have a different configuration, we can probably get rid of these concerns.

Thoughts ?


was (Author: varun_saxena):
Is this the right thing to do ?

NM whitelist configuration is used to allow containers to override NM's default environment.
Let us assume there is an environment variable DUMMY_ENV. Assume in some deployment, for some
jobs, its set in the ContainerLaunchContext and for all other jobs we want a default which
can be set in NM environment.
Before the change in the patch, we could have set DUMMY_ENV in NM whitelist and NM would have
picked up value for this environment variable from container launch context (if available)
otherwise taken it from NM environment.
After the change in this patch, NM would keep the value coming from container launch context
as default and use that only if the environment variable on NM is not set.

This is a departure from current behavior. Probably you can argue none of us are using it
this way but its still a change in behavior.
We have done the same for HADOOP_YARN_HOME, HADOOP_COMMON_HOME, etc. in YARN-3853 but these
are well known environment variables so probably nobody uses them the way I mentioned above.
So its safe to use them this way. If we were to have a different configuration, we can probably
get rid of these concerns.

Thoughts ?

> Allow all nm-whitelist-env to get overridden during launch
> ----------------------------------------------------------
>
>                 Key: YARN-5877
>                 URL: https://issues.apache.org/jira/browse/YARN-5877
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Bibin A Chundatt
>            Assignee: Bibin A Chundatt
>         Attachments: Dockerfile, YARN-5877.0001.patch, YARN-5877.0002.patch, bootstrap.sh,
yarn-site.xml
>
>
> As per the {{yarn.nodemanager.env-whitelist}} for the configured values should  containers
may override rather than use NodeManager's default.
> {code}
>   <property>
>     <description>Environment variables that containers may override rather than
use NodeManager's default.</description>
>     <name>yarn.nodemanager.env-whitelist</name>
>     <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME</value>
>   </property>
> {code}
> But only the following containers can override
> {code}
>     whitelist.add(ApplicationConstants.Environment.HADOOP_YARN_HOME.name());
>     whitelist.add(ApplicationConstants.Environment.HADOOP_COMMON_HOME.name());
>     whitelist.add(ApplicationConstants.Environment.HADOOP_HDFS_HOME.name());
>     whitelist.add(ApplicationConstants.Environment.HADOOP_CONF_DIR.name());
>     whitelist.add(ApplicationConstants.Environment.JAVA_HOME.name());
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message