hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Badger (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7677) HADOOP_CONF_DIR should not be automatically put in task environment
Date Wed, 03 Jan 2018 20:29:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310247#comment-16310247

Eric Badger commented on YARN-7677:

[~eyang], it doesn't necessarily need to be separate hadoop clusters. It could just be a node
where the NM runs on the bare metal host and the tasks run in docker containers. In that case,
they would need to know where {{HADOOP_CONF_DIR}} is. Since the docker image is completely
separate from the host layout, we can't assume that hadoop is going to be put in the same
place. {{HADOOP_CONF_DIR}} isn't getting bind-mounted into the container, so the only way
this would even work is by a happy coincidence and/or planning the layout of the image to
match that of the host. But that coupling is certainly not necessary and the docker image
is the one that actually knows where {{HADOOP_CONF_DIR}} is located. The nodemanager knows
where its {{HADOOP_CONF_DIR}} is located, but that is on the host, not in the docker container.

And again, since {{HADOOP_CONF_DIR}} is in the default env whitelist, the behavior here will
only change if you explicitly change the env whitelist and remove it. So I believe the impact
here to be fairly low. Regardless, I don't think it's correct for the NM to be defining the
layout of the docker image (i.e. where {{HADOOP_CONF_DIR}} has to be located). 

> HADOOP_CONF_DIR should not be automatically put in task environment
> -------------------------------------------------------------------
>                 Key: YARN-7677
>                 URL: https://issues.apache.org/jira/browse/YARN-7677
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Eric Badger
>            Assignee: Eric Badger
> Currently, {{HADOOP_CONF_DIR}} is being put into the task environment whether it's set
by the user or not. It completely bypasses the whitelist and so there is no way for a task
to not have {{HADOOP_CONF_DIR}} set. This causes problems in the Docker use case where Docker
containers will set up their own environment and have their own {{HADOOP_CONF_DIR}} preset
in the image itself. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message