hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sangjin Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4789) Provide helpful exception for non-PATH-like conflict with admin.user.env
Date Fri, 11 Mar 2016 18:33:39 GMT

    [ https://issues.apache.org/jira/browse/YARN-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15191345#comment-15191345

Sangjin Lee commented on YARN-4789:

Thanks for filing the issue and the patch [~jira.shegalov]! This definitely is a bug.

I also find the descriptions of the admin env properties very confusing:
  Expert: Additional execution environment entries for
  map and reduce task processes. This is not an additive property.
  You must preserve the original value if you want your map and
  reduce tasks to have access to native libraries (compression, etc).
  When this value is empty, the command to set execution
  envrionment will be OS dependent:
  For linux, use LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native.
  For windows, use PATH = %PATH%;%HADOOP_COMMON_HOME%\\bin.
  <description> Environment variables for the MR App Master
  processes for admin purposes. These values are set first and can be
  overridden by the user env (yarn.app.mapreduce.am.env) Example :
  1) A=foo  This will set the env variable A to foo
  2) B=$B:c This is inherit app master's B env variable.

The statement about it being non-additive is confusing, as for PATH-like variables they are
very much additive. Also, {{yarn.app.mapreduce.am.admin.user.env}} says 
bq. These values are set first and can be overridden by the user env (yarn.app.mapreduce.am.env)

So it seems to me that we should revisit these descriptions and improve them, especially if
we are going to reject user values for non-PATH environment variables.

I only took a quick look at the patch, but it appears the exception will be thrown every time
{{Apps.addToEnvironment()}} is called for a non-PATH environment variable? That would cause
exceptions for more types of situations than we're targeting, right? For example, I suspect
even the attempt to set {{yarn.app.mapreduce.am.admin.user.env}} would fail:

    // Setup the environment variables for Admin first
            MRJobConfig.DEFAULT_MR_AM_ADMIN_USER_ENV), conf);

I think it'd be better to target this change more narrowly.

> Provide helpful exception for non-PATH-like conflict with admin.user.env
> ------------------------------------------------------------------------
>                 Key: YARN-4789
>                 URL: https://issues.apache.org/jira/browse/YARN-4789
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 2.7.2
>            Reporter: Gera Shegalov
>            Assignee: Gera Shegalov
>         Attachments: YARN-4789.001.patch
> Environment variables specified in mapreduce.admin.user.env are supposed to be paths
(class, shell, library) and they can be merged with the user-provided values. However, it's
also possible that the cluster admins specify some non-PATH-like variable such as JAVA_HOME.
In this case if there is the same variable provided by the user, we'll get a concatenation
that does not make sense and is difficult to debug.

This message was sent by Atlassian JIRA

View raw message