hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2748) [MR-279] NM should pass a whitelisted environmental variables to the container
Date Fri, 29 Jul 2011 05:00:10 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13072680#comment-13072680

Vinod Kumar Vavilapalli commented on MAPREDUCE-2748:

Not that because of this bug, the NM's environment is not passed to streaming/pipes at all.
So the following commands will not work in YARN+MR.

    $HADOOP_COMMON_HOME//bin/hadoop --config $HADOOP_CONF_DIR jar $HADOOP_STREAMING_JAR -input
input.txt -mapper '$HADOOP_COMMON_HOME/bin/hadoop dfs -ls /'  -reducer NONE  -output Output

The failure is because HADOOP_COMMON_HOME is not known to the streaming process. Instead the
following will work, essentially you should expand HADOOP_COMMON_HOME yourselves for now:

    $HADOOP_COMMON_HOME//bin/hadoop --config $HADOOP_CONF_DIR jar $HADOOP_STREAMING_JAR -input
input.txt -mapper '/path/to/common/home/bin/hadoop dfs -ls /'  -reducer NONE  -output Output

> [MR-279] NM should pass a whitelisted environmental variables to the container 
> -------------------------------------------------------------------------------
>                 Key: MAPREDUCE-2748
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2748
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Vinod Kumar Vavilapalli
>             Fix For: 0.23.0
> This is similar to [MAPREDUCE-103] . We should pass a whitelisted set of environment
variables from NM env to the container. By default, we should pass HADOOP_* variables. This
can be a simple configuration key that NodeManager reads.
> Today, we already either pass the following correctly or assume that it works but doesn't
>  - YARN_HOME: ContainerLaunch#writeLaunchEnv
>  - HADOOP_CLIENT_OPTS: MapReduceChildJVM#setVMEnv
>  - JAVA_HOME: TaskAttemptImpl#createContainerLaunchContext - Works by shell-expansion.
>  - LD_LIBRARY_PATH: Assumed to work via shell-expansion but doesn't.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message