hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2625) Problems with CLASSPATH in Job Submission REST API
Date Tue, 23 Dec 2014 22:33:14 GMT

    [ https://issues.apache.org/jira/browse/YARN-2625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14257618#comment-14257618
] 

Vinod Kumar Vavilapalli commented on YARN-2625:
-----------------------------------------------

bq.  If the default ever changes, the code would need to change.
The way the layering works, YARN doesn't know whether the app is JVM based or not (though
predominant apps that exist today are all JVM based). Given this, we chose to introduce this
yarn.application.classpath as a _simplifying_ configuration property that apps can use. So
far, it didn't have any semantic meaning to the platform and it is important to keep it so
to keep it so. I think the right solution is for us to introduce some sort of a JAVA universe
as a API, we can auto-expand it the way you are asking only if the app specifically says it
is a JAVA universe.

bq. For example, if I had hardcoded the CLASSPATH value to the default of "$HADOOP_CONF_DIR,
$HADOOP_COMMON_HOME/share/hadoop/common/, $HADOOP_COMMON_HOME/share/hadoop/common/lib/, $HADOOP_HDFS_HOME/share/hadoop/hdfs/,
$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/, $HADOOP_YARN_HOME/share/hadoop/yarn/, $HADOOP_YARN_HOME/share/hadoop/yarn/lib/"
the classpath passed to the application master is ":/share/hadoop/common/:/share/hadoop/common/lib/:/share/hadoop/hdfs/:/share/hadoop/hdfs/lib/:/share/hadoop/yarn/:/share/hadoop/yarn/lib/"
This is a bug. NodeManager by default exports these environment variables and so the expansion
should happen automatically.

Are you using MR jobs? If so, once your clusters start moving to using MR tarballs through
distributed-cache scheme (MAPREDUCE-4421), the classpath dictated by mapreduce.application.classpath
will simply be a reflection of the directory hierarchy in the tarball and is completely decoupled
from the cluster-install or the environment variables.

> Problems with CLASSPATH in Job Submission REST API
> --------------------------------------------------
>
>                 Key: YARN-2625
>                 URL: https://issues.apache.org/jira/browse/YARN-2625
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: api
>    Affects Versions: 2.5.1
>            Reporter: Doug Haigh
>
> There are a couple of issues I have found specifying the CLASSPATH environment variable
using the REST API.
> 1) In the Java client, the CLASSPATH environment is usually made up of either the value
of the yarn.application.classpath in yarn-site.xml value or the default YARN classpath value
as defined by YarnConfiguration.DEFAULT_YARN_CROSS_PLATFORM_APPLICATION_CLASSPATH. REST API
consumers have no method of telling the resource manager to use the default unless they hardcode
the default value themselves. If the default ever changes, the code would need to change.

> 2) If any environment variables are used in the CLASSPATH environment 'value' field,
they are evaluated when the values are NULL resulting in bad values in the CLASSPATH. For
example, if I had hardcoded the CLASSPATH value to the default of "$HADOOP_CONF_DIR, $HADOOP_COMMON_HOME/share/hadoop/common/*,
$HADOOP_COMMON_HOME/share/hadoop/common/lib/*, $HADOOP_HDFS_HOME/share/hadoop/hdfs/*, $HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,
$HADOOP_YARN_HOME/share/hadoop/yarn/*, $HADOOP_YARN_HOME/share/hadoop/yarn/lib/*" the classpath
passed to the application master is ":/share/hadoop/common/*:/share/hadoop/common/lib/*:/share/hadoop/hdfs/*:/share/hadoop/hdfs/lib/*:/share/hadoop/yarn/*:/share/hadoop/yarn/lib/*"
> These two problems require REST API consumers to always have the fully resolved path
defined in the yarn.application.classpath value. If the property is missing or contains environment
varaibles, the application created by the REST API will fail due to the CLASSPATH being incorrect.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message