hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Subru Krishnan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
Date Sat, 04 Jun 2016 01:28:59 GMT

    [ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15315223#comment-15315223

Subru Krishnan commented on YARN-5139:

+1 for the proposal. We have had multiple discussions offline about this with [~leftnoteasy],
[~kasha], [~vinodkv], [~asuresh], etc.... Thanks to [~leftnoteasy] for initiating this. 

Are you planning to add _node labels_  (actually the GUTS API) to {{NodeCandidates}} and {{SchedulerNodesScorer}}
for filtering? 

I am also in agreement with [~kasha], [~asuresh] that we should take this opportunity to consider
separating *Cluster State* from *Application Scheduling*. This should allows to scale/optimize
them individually.

At the risk of over-simplification, we can potentially evolve the current scheduler specific
policies to be limited to the ordering of applications (a la priority heap sort algorithm).

I am happy to contribute, let me know if you need any help. 

> [Umbrella] Move YARN scheduler towards global scheduler
> -------------------------------------------------------
>                 Key: YARN-5139
>                 URL: https://issues.apache.org/jira/browse/YARN-5139
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: wip-1.YARN-5139.patch, wip-2.YARN-5139.patch
> Existing YARN scheduler is based on node heartbeat. This can lead to sub-optimal decisions
because scheduler can only look at one node at the time when scheduling resources.
> Pseudo code of existing scheduling logic looks like:
> {code}
> for node in allNodes:
>    Go to parentQueue
>       Go to leafQueue
>         for application in leafQueue.applications:
>            for resource-request in application.resource-requests
>               try to schedule on node
> {code}
> Considering future complex resource placement requirements, such as node constraints
(give me "a && b || c") or anti-affinity (do not allocate HBase regionsevers and Storm
workers on the same host), we may need to consider moving YARN scheduler towards global scheduling.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message