hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Badger (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7677) HADOOP_CONF_DIR should not be automatically put in task environment
Date Wed, 03 Jan 2018 16:48:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309906#comment-16309906

Eric Badger commented on YARN-7677:

[~eyang], the docker container file system layout doesn't have to be the same as the host.
It's entirely possible for an image to have {{HADOOP_CONF_DIR}} set to something like {{/home/conf}},
while the host has it in {{/tmp/conf}}. In the current implementation, there is no way for
{{HADOOP_CONF_DIR}} to be set correctly in this case without the user explicitly setting it
with their job. Even if you set up {{HADOOP_CONF_DIR}} in the image as an environment variable,
it will be overridden by the container startup script, since {{HADOOP_CONF_DIR}} is being
passed in by default regardless. In the case where a user doesn't set {{HADOOP_CONF_DIR}}
explicitly, it makes sense that the docker image will know where to set it (via ENV), while
the NM will not, since their layouts are not necessarily the same.

> HADOOP_CONF_DIR should not be automatically put in task environment
> -------------------------------------------------------------------
>                 Key: YARN-7677
>                 URL: https://issues.apache.org/jira/browse/YARN-7677
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Eric Badger
>            Assignee: Eric Badger
> Currently, {{HADOOP_CONF_DIR}} is being put into the task environment whether it's set
by the user or not. It completely bypasses the whitelist and so there is no way for a task
to not have {{HADOOP_CONF_DIR}} set. This causes problems in the Docker use case where Docker
containers will set up their own environment and have their own {{HADOOP_CONF_DIR}} preset
in the image itself. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message