hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1360) Rework Distributed Shell to be a better model of how people should write YARN applications
Date Tue, 29 Oct 2013 15:04:32 GMT

    [ https://issues.apache.org/jira/browse/YARN-1360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13808058#comment-13808058

Steve Loughran commented on YARN-1360:

I'm not volunteering to do this, though I now know what it should look like -having just reworked
my YARN application to the extent it no longer resembles the Distributed Shell -and instead
has all the features I've just listed above as requirements.

The problem with the dist shell today is that it is "the simple" example, with mapreduce being
way to complicated to go near. Yet a lot of the "real" requirements of a YARN app lurk in
the MR code, not in distributed shell, while the architecture of the shell is exactly what
you don't want for testing and maintenance - everything in the AM class.

> Rework Distributed Shell to be a better model of how people should write YARN applications
> ------------------------------------------------------------------------------------------
>                 Key: YARN-1360
>                 URL: https://issues.apache.org/jira/browse/YARN-1360
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: applications/distributed-shell
>    Affects Versions: 2.2.0
>            Reporter: Steve Loughran
> Distributed Shell works as an example, but it's not the right architecture for something
you'd want in production -instead its design runs a risk of setting a  bad example for others
to follow.
> Instead it should
> * be decomposed into a set of services each with their own responsibilities
> * split the 'model' of its cluster into its own classes, a model that can then be unit
tested outside of the AM
> * factor out all container launching into its own service -and use a thread pool to avoid
scalability limits
> * provide a demonstration (classic) RPC service to show how to implement this
> * tests

This message was sent by Atlassian JIRA

View raw message