hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jim Brennan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7677) Docker image cannot set HADOOP_CONF_DIR
Date Mon, 19 Feb 2018 14:57:02 GMT

    [ https://issues.apache.org/jira/browse/YARN-7677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369205#comment-16369205
] 

Jim Brennan commented on YARN-7677:
-----------------------------------

Agreed. Without this patch, the ordering of variables in the launch_container.sh script is
random - it is just iterating a hashmap, so there is no guarantee of the order for any of
the variables.

With this patch, the following order is imposed:
 # Whitelisted variables not overridden by user, in the order they are listed in NM_ENV_WHITELIST
 # Variables set explicitly by the NM, in the order they are written in code.
 # User-defined variables, with no ordering guarantee.

[~rohithsharma], it looks like you are running into the last category, where CLASSPATH and
HADOOP_MAPRED_HOME are both user-defined.   As [~jlowe] said, it looks like we will have
to address YARN-5714 so that we can guarantee the order of the user-defined variables before
we can put this patch in.

Thanks for reporting the problem - and thanks [~jlowe] for reverting this one.

> Docker image cannot set HADOOP_CONF_DIR
> ---------------------------------------
>
>                 Key: YARN-7677
>                 URL: https://issues.apache.org/jira/browse/YARN-7677
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 3.0.0
>            Reporter: Eric Badger
>            Assignee: Jim Brennan
>            Priority: Major
>         Attachments: YARN-7677.001.patch, YARN-7677.002.patch, YARN-7677.003.patch, YARN-7677.004.patch,
YARN-7677.005.patch
>
>
> Currently, {{HADOOP_CONF_DIR}} is being put into the task environment whether it's set
by the user or not. It completely bypasses the whitelist and so there is no way for a task
to not have {{HADOOP_CONF_DIR}} set. This causes problems in the Docker use case where Docker
containers will set up their own environment and have their own {{HADOOP_CONF_DIR}} preset
in the image itself. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message