hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
Date Tue, 24 May 2016 22:08:12 GMT

    [ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15299024#comment-15299024
] 

Karthik Kambatla edited comment on YARN-5139 at 5/24/16 10:07 PM:
------------------------------------------------------------------

[~leftnoteasy] - thanks for filing this.

In our experience with continuous scheduling, this approach will help for faster scheduling
especially for high-priority applications. That said, once you gain more experience with your
prototype, can we actually decide on the right abstractions so the schedulers share more code
in the new world. 


was (Author: kasha):
[~leftnoteasy] - thanks for filing this.

In our experience with continuous scheduling, this approach will help for faster scheduling
especially for high-priority applications. That said, once you gain more experience with your
prototype, can we actually decide on the right abstracts so in the new world the schedulers
share more code. 

> [Umbrella] Move YARN scheduler towards global scheduler
> -------------------------------------------------------
>
>                 Key: YARN-5139
>                 URL: https://issues.apache.org/jira/browse/YARN-5139
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: wip-1.YARN-5139.patch
>
>
> Existing YARN scheduler is based on node heartbeat. This can lead to sub-optimal decisions
because scheduler can only look at one node at the time when scheduling resources.
> Pseudo code of existing scheduling logic looks like:
> {code}
> for node in allNodes:
>    Go to parentQueue
>       Go to leafQueue
>         for application in leafQueue.applications:
>            for resource-request in application.resource-requests
>               try to schedule on node
> {code}
> Considering future complex resource placement requirements, such as node constraints
(give me "a && b || c") or anti-affinity (do not allocate HBase regionsevers and Storm
workers on the same host), we may need to consider moving YARN scheduler towards global scheduling.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message