reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashvin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (REEF-1274) Add validation to YarnConfigurationConstructor
Date Wed, 23 Mar 2016 21:00:28 GMT

    [ https://issues.apache.org/jira/browse/REEF-1274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15209160#comment-15209160
] 

Ashvin commented on REEF-1274:
------------------------------

I think to a user {MiniYarnCluster} is similar to full YARN setup. It is an in-process YARN
cluster. Even the default config names and values are the same.  
Example code to start the test cluster: {cluster = new MiniYARNCluster(...).start()}. 

Once the test cluster starts, {YarnConfiguration} in the client can be constructed like this:
{new YarnConfiguration().set( "yarn.resourcemanager.address", "hostname:" + cluster.getPort()
)}. Subsequently yarn client connects to the test yarn cluster.

If {YarnConfigurationConstructor} could be provided with the RM address, then it could prepare
{YarnConfiguration}. Alternatively {YarnDriverConfiguration} could allow binding of a custom
{YarnConfigurationConstructor}. Given this, I think a new runtime for test cluster is an overhead.

> Add validation to YarnConfigurationConstructor
> ----------------------------------------------
>
>                 Key: REEF-1274
>                 URL: https://issues.apache.org/jira/browse/REEF-1274
>             Project: REEF
>          Issue Type: New Feature
>          Components: REEF-Runtime-YARN
>            Reporter: Markus Weimer
>
> A frequent problem REEF applications encounter is a broken or unavailable YARN Configuration
of the cluster. REEF can help with the diagnose of such issues as we always load / instantiate
the YARN Configuration via {{o..r.runtime.yarn.util.YarnConfigurationConstructor}}. In that
class, we can validate the YARN configuration and report WARNINGs into the log if something
is odd. For example:
>   * Is the RM IP set and reachable?
>   * Is the classpath set and non-empty? If not, are the environment variables set?
>   * Can a `FileSystem` instance be created using that configuration?
> There are probably more tests we can perform as we hear from our users.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message