hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jim Brennan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7677) Docker image cannot set HADOOP_CONF_DIR
Date Mon, 26 Feb 2018 15:29:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16377034#comment-16377034
] 

Jim Brennan commented on YARN-7677:
-----------------------------------

Now that YARN-5714 has been resolved, we have two options for resolving this Jira. Environment
variables now go through a dependency sort before being written to the launch_container.sh
script. So we need to decide whether we want to retain the ordering of the 3 categories of
environment variables from the last patch for this Jira, or just do the minimal changes to
address the original issue- allowing (docker) container images to override the whitelisted
variables.

So the two options are:

Minimal changes:
 * Remove the explicit setting of HADOOP_CONF_DIR, and treat it like other whitelisted variables.
 * Whitelisted variables not overridden by the user are written first, in the order they are
listed in the NM_ENV_WHITELIST, using the {{var:-default}} variable expansion syntax
 * All other variables are then written in dependency sorted order.

Category sorted order:
 * Remove the explicit setting of HADOOP_CONF_DIR, and treat it like other whitelisted variables.
 * Whitelisted variables not overridden by the user are written first, in the order they are
listed in the NM_ENV_WHITELIST, using the {{var:-default}} variable expansion syntax.
 * Then write variables set explicitly by the NM, in the order they are written in code.
 * Finally, write user-defined variables in dependency sorted order.

The main difference is that the second approach ensures all of the explicitly set NM variables
are always written before all user variables, in a consistent order.   I do like seeing
that consistency when looking at the scripts, but I'm not sure it's required, just nice.

[~jlowe], [~ebadger], [~shanekumpf@gmail.com], [~billie.rinaldi], please let me know if you
have a preference.

> Docker image cannot set HADOOP_CONF_DIR
> ---------------------------------------
>
>                 Key: YARN-7677
>                 URL: https://issues.apache.org/jira/browse/YARN-7677
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 3.0.0
>            Reporter: Eric Badger
>            Assignee: Jim Brennan
>            Priority: Major
>         Attachments: YARN-7677.001.patch, YARN-7677.002.patch, YARN-7677.003.patch, YARN-7677.004.patch,
YARN-7677.005.patch
>
>
> Currently, {{HADOOP_CONF_DIR}} is being put into the task environment whether it's set
by the user or not. It completely bypasses the whitelist and so there is no way for a task
to not have {{HADOOP_CONF_DIR}} set. This causes problems in the Docker use case where Docker
containers will set up their own environment and have their own {{HADOOP_CONF_DIR}} preset
in the image itself. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message