hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2915) Enable YARN RM scale out via federation using multiple RM's
Date Wed, 16 Mar 2016 19:15:33 GMT

    [ https://issues.apache.org/jira/browse/YARN-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15197956#comment-15197956

Vinod Kumar Vavilapalli commented on YARN-2915:

One thing that occurred to me in an offline conversation with [~subru] and [~leftnoteasy]
is about the modeling of queues and their shares in different sub-clusters.

As seems to be already proposed, it is very desirable to have a unified *logic queues* that
are applicable across all sub-clusters.

With unified logical queues, looks like there are some proposals for ways of how resources
can get sub-divided amongst different sub-clusters. But to me, they already map to an existing
concept in YARN - *Node Partitions* / node-labels !

Essentially you have *one YARN cluster* -> *multiple sub-clusters* -> *each sub-cluster
with multiple node-partitions*. This can further be extended to more levels. For e.g. we can
unify rack also under the same concept.

The advantage of unifying this with node-partitions is that we can have
 - one single administrative view philosophy of sub-clusters, node-partitions, racks etc
 - unified configuration mechanisms: Today we support centralized and distributed node-partition
mechanisms, exclusive / non-exclusive access etc.
 - unified queue-sharing models - today we already can assign X% of a node-partition to a
queue. This way we can, again, reuse existing concepts, mental models and allocation policies
- instead of creating specific policies for sub-cluster sharing like the user-based share
that is proposed.

We will have to dig deeper into the details, but it seems to me that node-partition and sub-cluster
are equivalence classes except for the fact that two sub-clusters report to two different
RMs (physically / implementation wise) which isn't the case today with node-partitions.

Thoughts? /cc [~curino] [~chris.douglas]

> Enable YARN RM scale out via federation using multiple RM's
> -----------------------------------------------------------
>                 Key: YARN-2915
>                 URL: https://issues.apache.org/jira/browse/YARN-2915
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: nodemanager, resourcemanager
>            Reporter: Sriram Rao
>            Assignee: Subru Krishnan
>         Attachments: FEDERATION_CAPACITY_ALLOCATION_JIRA.pdf, Federation-BoF.pdf, Yarn_federation_design_v1.pdf,
> This is an umbrella JIRA that proposes to scale out YARN to support large clusters comprising
of tens of thousands of nodes.   That is, rather than limiting a YARN managed cluster to about
4k in size, the proposal is to enable the YARN managed cluster to be elastically scalable.

This message was sent by Atlassian JIRA

View raw message