hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun Suresh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4902) [Umbrella] Generalized and unified scheduling-strategies in YARN
Date Wed, 13 Apr 2016 17:34:25 GMT

    [ https://issues.apache.org/jira/browse/YARN-4902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15239662#comment-15239662

Arun Suresh commented on YARN-4902:

Does it make sense to introduce the concept of *AffinityGroups* / *Anti-AffinityGroups* for
the purpose of simplifying affinity/anti-affinity  ?
Essentially, just like we use the Reservation API to request a reservation Id, and then include
the Id in subsequent requests,
we could expose an API that allows one to request an *(Anti)AffinityGroups* (consisting of
N machine / totally X resources etc.) which inturn returns a *groupId*. (This could internally
would dynamically tag / label a group of Machines/Nodes with the id, but I guess that is an
implementation detail.)

Subsequent allocation requests can include this *groupId* and based on the policy, The scheduler
will try to schedule on the same group of machines or different machines.

Another though is, an affinity group may be an Allocation that can be used/shared by multiple
applications. Once we de-link Allocations from containers, It may be possible to have an uber-application
(or maybe a component in the RM itself) lease Allocations only and grant the right to start
containers against these allocations to other applications. A group of allocations across
multiple machines can constitute an anti-affinity group, and we can introduce policies that
allow subsequent apps to, say, start only 1 container of each role/type on 1 allocation within
a group for eg.

I feel that allowing users to organize deployments along the lines of affinity/anti-affinity
groups (failure domains might also be something similar) is more manageable than allowing
users to specify affinity and anti-affinity constraints with respect to an already deployed

> [Umbrella] Generalized and unified scheduling-strategies in YARN
> ----------------------------------------------------------------
>                 Key: YARN-4902
>                 URL: https://issues.apache.org/jira/browse/YARN-4902
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Wangda Tan
>         Attachments: Generalized and unified scheduling-strategies in YARN -v0.pdf
> Apache Hadoop YARN's ResourceRequest mechanism is the core part of the YARN's scheduling
API for applications to use. The ResourceRequest mechanism is a powerful API for applications
(specifically ApplicationMasters) to indicate to YARN what size of containers are needed,
and where in the cluster etc.
> However a host of new feature requirements are making the API increasingly more and more
complex and difficult to understand by users and making it very complicated to implement within
the code-base.
> This JIRA aims to generalize and unify all such scheduling-strategies in YARN.

This message was sent by Atlassian JIRA

View raw message