reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashvin (JIRA)" <>
Subject [jira] [Commented] (REEF-1274) Add validation to YarnConfigurationConstructor
Date Wed, 23 Mar 2016 18:04:25 GMT


Ashvin commented on REEF-1274:

Early detection and reporting will be very useful. I think the use cases listed above will
cover most scenarios. I have one other use cases. I am not sure if the use cases need separate
discussion/jira threads. Many yarn app developers use {{MiniYarnCluster}} to unit test the
application. In this case the RM address address in generated at runtime. Which means the
address is not part of a config file in classpath. I am wondering if the test RM address could
be provided as a configuration to {{YarnConfigurationConstructor}}.

> Add validation to YarnConfigurationConstructor
> ----------------------------------------------
>                 Key: REEF-1274
>                 URL:
>             Project: REEF
>          Issue Type: New Feature
>          Components: REEF-Runtime-YARN
>            Reporter: Markus Weimer
> A frequent problem REEF applications encounter is a broken or unavailable YARN Configuration
of the cluster. REEF can help with the diagnose of such issues as we always load / instantiate
the YARN Configuration via {{o..r.runtime.yarn.util.YarnConfigurationConstructor}}. In that
class, we can validate the YARN configuration and report WARNINGs into the log if something
is odd. For example:
>   * Is the RM IP set and reachable?
>   * Is the classpath set and non-empty? If not, are the environment variables set?
>   * Can a `FileSystem` instance be created using that configuration?
> There are probably more tests we can perform as we hear from our users.

This message was sent by Atlassian JIRA

View raw message