hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sunil G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5877) Allow all nm-whitelist-env to get overridden during launch
Date Wed, 07 Dec 2016 06:38:01 GMT

    [ https://issues.apache.org/jira/browse/YARN-5877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15727893#comment-15727893

Sunil G commented on YARN-5877:

We have checked various cases for this issue in detail and had some offline discussions with
[~vvasudev] and [~bibinchundatt] also.

Now coming to the issue mentioned. MAPREDUCE-6704 has come in to a common consensus that HADOOP_MAPRED_HOME
could be published with nm-white-list so that even if this variable is not present in env,
we can take the configured value. This will solve the container launch problem mentioned there.

As of today, if we publish HADOOP_MAPRED_HOME through nm-whitelist-env, it will look like
below in launch_container.sh.
{{export HADOOP_MAPRED_HOME="/home/hadoopbuild"}}

As mentioned above, this will be fine in case of normal container launch. But in docker scenario,
the path value given may not be valid. In such case, we will see the failure as mentioned
in this jira. Ideally if this environment variable were published as whitelist env, then it
could try to take from env itself first.

export HADOOP_MAPRED_HOME=${HADOOP_MAPRED_HOME:-"/home/hadoopbuild"}

Here container will try to get the value from *HADOOP_MAPRED_HOME* if its available in env
first. If not, it will take value which is given as "/home/hadoopbuild". In case of docker,
this value was set in its system env itself (i think Bibin used sequence iq docker image here
which had this value).

So in conclusion, all variables mentioned in {{yarn.nodemanager.env-whitelist}} could be whitelisted.
This can solve the problem. If some docker images doesn't have this env variable set in its
env, and if the default path to HADOOP_MAPRED_HOME is also invalid, then user has to set correct
mapred home path in {{yarn.app.mapreduce.am.env}}.

I think YARN-5877.0002.patch is good one to go. [~bibinchundatt] could you rebase the patch
and give a latest version number to compile against trunk. If possible, could you please share
test results with this latest patch too in docker.

> Allow all nm-whitelist-env to get overridden during launch
> ----------------------------------------------------------
>                 Key: YARN-5877
>                 URL: https://issues.apache.org/jira/browse/YARN-5877
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Bibin A Chundatt
>            Assignee: Bibin A Chundatt
>         Attachments: Dockerfile, YARN-5877.0001.patch, YARN-5877.0002.patch, YARN-5877.0003.patch,
bootstrap.sh, yarn-site.xml
> As per the {{yarn.nodemanager.env-whitelist}} for the configured values should  containers
may override rather than use NodeManager's default.
> {code}
>   <property>
>     <description>Environment variables that containers may override rather than
use NodeManager's default.</description>
>     <name>yarn.nodemanager.env-whitelist</name>
>   </property>
> {code}
> But only the following containers can override
> {code}
>     whitelist.add(ApplicationConstants.Environment.HADOOP_YARN_HOME.name());
>     whitelist.add(ApplicationConstants.Environment.HADOOP_COMMON_HOME.name());
>     whitelist.add(ApplicationConstants.Environment.HADOOP_HDFS_HOME.name());
>     whitelist.add(ApplicationConstants.Environment.HADOOP_CONF_DIR.name());
>     whitelist.add(ApplicationConstants.Environment.JAVA_HOME.name());
> {code}

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message