hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
Date Tue, 24 May 2016 23:34:13 GMT

    [ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15299146#comment-15299146

Wangda Tan commented on YARN-5139:

[~asuresh], [~kasha],

The major reason of adding the {{NodeCandidates}} instead of using APIs in ClusterNodeTracker
is we can cache results instead of doing O(n) filtering at ClusterNodeTracker. I'm not sure
if we can merge the two filters, it will become more clear while doing the POC.
Similarly, using {{SchedulerNodesScorer}} instead of ClusterNodeTracker.sortedNodeList is
we don't have to do full sort of the list.

For the constraints between applications, YARN-4902 can definitely cover it. However, I'm
not sure what's the best way to describe it before we have YARN-4902. I don't feel strongly
that we shouldn't put it into resource request, contradicting constraints should be detectable.
We can continue discuss about it in YARN-1042.

> [Umbrella] Move YARN scheduler towards global scheduler
> -------------------------------------------------------
>                 Key: YARN-5139
>                 URL: https://issues.apache.org/jira/browse/YARN-5139
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: wip-1.YARN-5139.patch
> Existing YARN scheduler is based on node heartbeat. This can lead to sub-optimal decisions
because scheduler can only look at one node at the time when scheduling resources.
> Pseudo code of existing scheduling logic looks like:
> {code}
> for node in allNodes:
>    Go to parentQueue
>       Go to leafQueue
>         for application in leafQueue.applications:
>            for resource-request in application.resource-requests
>               try to schedule on node
> {code}
> Considering future complex resource placement requirements, such as node constraints
(give me "a && b || c") or anti-affinity (do not allocate HBase regionsevers and Storm
workers on the same host), we may need to consider moving YARN scheduler towards global scheduling.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message