hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Subru Krishnan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-2915) Enable YARN RM scale out via federation using multiple RM's
Date Mon, 30 Oct 2017 19:32:02 GMT

     [ https://issues.apache.org/jira/browse/YARN-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Subru Krishnan updated YARN-2915:
---------------------------------
    Release Note: A federation-based approach to transparently scale a single YARN cluster
to tens of thousands of nodes, by federating multiple YARN standalone clusters.The proposed
approach is to divide a large (10-100k nodes) cluster into smaller units called sub-clusters,
each with its own YARN RM and compute nodes. The federation system will stitch these sub-clusters
together and make them appear as one large YARN cluster to the applications. The applications
running in this federated environment will see a single massive YARN cluster and will be able
to schedule tasks on any node of the federated cluster. Under the hood, the federation system
will negotiate with sub-clusters resource managers and provide resources to the application.
The goal is to allow an individual job to “span” sub-clusters seamlessly.  (was: A federation-based
approach to transparently scale a single YARN cluster to tens of thousands of nodes, by federating
multiple YARN standalone clusters)

> Enable YARN RM scale out via federation using multiple RM's
> -----------------------------------------------------------
>
>                 Key: YARN-2915
>                 URL: https://issues.apache.org/jira/browse/YARN-2915
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: nodemanager, resourcemanager
>            Reporter: Sriram Rao
>            Assignee: Subru Krishnan
>              Labels: federation
>             Fix For: 2.9.0, 3.0.0-beta1
>
>         Attachments: FEDERATION_CAPACITY_ALLOCATION_JIRA.pdf, Federation-BoF.pdf, YARN-Federation-Hadoop-Summit_final.pptx,
Yarn_federation_design_v1.pdf, federation-prototype.patch
>
>
> This is an umbrella JIRA that proposes to scale out YARN to support large clusters comprising
of tens of thousands of nodes.   That is, rather than limiting a YARN managed cluster to about
4k in size, the proposal is to enable the YARN managed cluster to be elastically scalable.
 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message