hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2915) Enable YARN RM scale out via federation using multiple RM's
Date Wed, 16 Mar 2016 19:15:33 GMT

    [ https://issues.apache.org/jira/browse/YARN-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15197956#comment-15197956
] 

Vinod Kumar Vavilapalli commented on YARN-2915:
-----------------------------------------------

One thing that occurred to me in an offline conversation with [~subru] and [~leftnoteasy]
is about the modeling of queues and their shares in different sub-clusters.

As seems to be already proposed, it is very desirable to have a unified *logic queues* that
are applicable across all sub-clusters.

With unified logical queues, looks like there are some proposals for ways of how resources
can get sub-divided amongst different sub-clusters. But to me, they already map to an existing
concept in YARN - *Node Partitions* / node-labels !

Essentially you have *one YARN cluster* -> *multiple sub-clusters* -> *each sub-cluster
with multiple node-partitions*. This can further be extended to more levels. For e.g. we can
unify rack also under the same concept.

The advantage of unifying this with node-partitions is that we can have
 - one single administrative view philosophy of sub-clusters, node-partitions, racks etc
 - unified configuration mechanisms: Today we support centralized and distributed node-partition
mechanisms, exclusive / non-exclusive access etc.
 - unified queue-sharing models - today we already can assign X% of a node-partition to a
queue. This way we can, again, reuse existing concepts, mental models and allocation policies
- instead of creating specific policies for sub-cluster sharing like the user-based share
that is proposed.

We will have to dig deeper into the details, but it seems to me that node-partition and sub-cluster
are equivalence classes except for the fact that two sub-clusters report to two different
RMs (physically / implementation wise) which isn't the case today with node-partitions.

Thoughts? /cc [~curino] [~chris.douglas]

> Enable YARN RM scale out via federation using multiple RM's
> -----------------------------------------------------------
>
>                 Key: YARN-2915
>                 URL: https://issues.apache.org/jira/browse/YARN-2915
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: nodemanager, resourcemanager
>            Reporter: Sriram Rao
>            Assignee: Subru Krishnan
>         Attachments: FEDERATION_CAPACITY_ALLOCATION_JIRA.pdf, Federation-BoF.pdf, Yarn_federation_design_v1.pdf,
federation-prototype.patch
>
>
> This is an umbrella JIRA that proposes to scale out YARN to support large clusters comprising
of tens of thousands of nodes.   That is, rather than limiting a YARN managed cluster to about
4k in size, the proposal is to enable the YARN managed cluster to be elastically scalable.
 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message