reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrey Meleshko <andr...@microsoft.com>
Subject YarnCalsspathProvider initialization: cluster problem?
Date Wed, 02 Nov 2016 16:02:24 GMT
We are facing what looks like cluster configuration problem
and I am hoping someone on the list can help with next investigation steps.

We see following error in driver logs: SEVERE: YarnConfiguration.YARN_APPLICATION_CLASSPATH
is empty. This indicates a broken cluster configuration

This is coming from YarnClasspathProvider initialization code, when the property value is
not present.
We've checked HADOOP_HOME variable and yarn-site.xml content: both seem to be correct.
Anyone knows how this property is initialized and what to check next in the cluster?

Related questions:

1)  DEFAULT_YARN_CROSS_PLATFORM_APPLICATION_CLASSPATH usage:
I can see that default classpath property is set to some value, but it is used only if YARN_APPLICATION_CLASSPATH
is present, and ignored otherwise.
Would it make sense to use default classpath in any case, if it's present?

YarnConfiguration.DEFAULT_YARN_CROSS_PLATFORM_APPLICATION_CLASSPATH is [{{HADOOP_CONF_DIR}}|{{HADOOP_COMMON_HOME}}/share/hadoop/common/*|{{HADOOP_COMMON_HOME}}/share/hadoop/common/lib/*|{{HADOOP_HDFS_HOME}}/share/hadoop/hdfs/*|{{HADOOP_HDFS_HOME}}/share/hadoop/hdfs/lib/*|{{HADOOP_YARN_HOME}}/share/hadoop/yarn/*|{{HADOOP_YARN_HOME}}/share/hadoop/yarn/lib/*]


2)  DEFAULT_YARN_APPLICATION_CLASSPATH usage:
This property is not used at all in YarnClasspathProvider.
I am not sure if it's initialized in my case, but it sounds like something YarnClasspathProvider
should fallback on if the YARN_APPICATION_CLASSPATH is not set.
Thank you,
Andrey

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message