hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varun Vasudev (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2625) Problems with CLASSPATH in Job Submission REST API
Date Mon, 06 Oct 2014 16:20:34 GMT

    [ https://issues.apache.org/jira/browse/YARN-2625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160442#comment-14160442
] 

Varun Vasudev commented on YARN-2625:
-------------------------------------

[~cdhaigh], you can fetch the config via http://<rm address>:<rm port>/conf. That
gives you the xml configuration for the RM. Can you extract yarn.application.classpath from
there and use it when submitting your app?

> Problems with CLASSPATH in Job Submission REST API
> --------------------------------------------------
>
>                 Key: YARN-2625
>                 URL: https://issues.apache.org/jira/browse/YARN-2625
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: api
>    Affects Versions: 2.5.1
>            Reporter: Doug Haigh
>
> There are a couple of issues I have found specifying the CLASSPATH environment variable
using the REST API.
> 1) In the Java client, the CLASSPATH environment is usually made up of either the value
of the yarn.application.classpath in yarn-site.xml value or the default YARN classpath value
as defined by YarnConfiguration.DEFAULT_YARN_CROSS_PLATFORM_APPLICATION_CLASSPATH. REST API
consumers have no method of telling the resource manager to use the default unless they hardcode
the default value themselves. If the default ever changes, the code would need to change.

> 2) If any environment variables are used in the CLASSPATH environment 'value' field,
they are evaluated when the values are NULL resulting in bad values in the CLASSPATH. For
example, if I had hardcoded the CLASSPATH value to the default of "$HADOOP_CONF_DIR, $HADOOP_COMMON_HOME/share/hadoop/common/*,
$HADOOP_COMMON_HOME/share/hadoop/common/lib/*, $HADOOP_HDFS_HOME/share/hadoop/hdfs/*, $HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,
$HADOOP_YARN_HOME/share/hadoop/yarn/*, $HADOOP_YARN_HOME/share/hadoop/yarn/lib/*" the classpath
passed to the application master is ":/share/hadoop/common/*:/share/hadoop/common/lib/*:/share/hadoop/hdfs/*:/share/hadoop/hdfs/lib/*:/share/hadoop/yarn/*:/share/hadoop/yarn/lib/*"
> These two problems require REST API consumers to always have the fully resolved path
defined in the yarn.application.classpath value. If the property is missing or contains environment
varaibles, the application created by the REST API will fail due to the CLASSPATH being incorrect.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message