hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jim Brennan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7677) Docker image cannot set HADOOP_CONF_DIR
Date Mon, 12 Feb 2018 20:35:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361418#comment-16361418
] 

Jim Brennan commented on YARN-7677:
-----------------------------------

I've put up another patch for this that addresses the ordering issues.

The new patch writes environment variables to the launch script in the following order:
 # Whitelisted variables that are coming from the NM environment (they were not set in the
ContainerLaunchContext).  These are written using the default value syntax, so that whitelisted
variables that are also set in the docker container will use the docker container setting. 
These are written in the order they are listed in the NM_ENV_WHITELIST yarn config.
 # Variables that are explicitly set by the nodemanager (mostly in ContainerLaunch.sanitizeEnv()). 
These are tracked so that they are written to the launch script in the order they are added
in code.
 # Remaining variables from the ContainerLaunchContext.  Note that these are still not ordered
- they are just written in hash-map order.  Changing these to ordered is covered by YARN-5714. 
This patch just ensures that they come after whitelisted and NM vars.

 A couple notes about this patch:
 * Variables that are listed in NM_ENV_WHITELIST, but are also set in the ContainerLaunchContext()
are written along with the rest of the ContainerLaunchContext environment variables (after
NM variables).
 * I am currently tracking as an NM variable the CLASSPATH that is touched in sanitizeWindowsEnv(),
but I think this is incorrect.  sanitizeWindowsEnv() only modifies CLASSPATH if it's already
in the ContainerLaunchContext().
 * The variables defined in NM_ADMIN_USER_ENV are tracked along with those set by NM.  I'm
not sure this is correct.

The patch includes a new test that verifies this ordering.

 

> Docker image cannot set HADOOP_CONF_DIR
> ---------------------------------------
>
>                 Key: YARN-7677
>                 URL: https://issues.apache.org/jira/browse/YARN-7677
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 3.0.0
>            Reporter: Eric Badger
>            Assignee: Jim Brennan
>            Priority: Major
>         Attachments: YARN-7677.001.patch, YARN-7677.002.patch, YARN-7677.003.patch
>
>
> Currently, {{HADOOP_CONF_DIR}} is being put into the task environment whether it's set
by the user or not. It completely bypasses the whitelist and so there is no way for a task
to not have {{HADOOP_CONF_DIR}} set. This causes problems in the Docker use case where Docker
containers will set up their own environment and have their own {{HADOOP_CONF_DIR}} preset
in the image itself. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message