hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
Date Wed, 24 Aug 2016 22:01:20 GMT

     [ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Wangda Tan updated YARN-5139:
    Attachment: wip-4.YARN-5139.patch

Also sharing updated prototype patch (wip-4).

It has changes: 
- Most logics described in the doc (v2), such as resource allocator / committer, placement
set, scorer, etc.
- Read/write lock optimizations for some of the scheduler classes (Major reason of the scary
patch size).

It should work properly for CapacityScheduler, I have updated the patch to make it work for
most integration unit tests, such as tests in TestCapacityScheduler / TestContainerAllocation
/ TestNodeLabelContainerAllocation, etc.

And please note that this patch doesn't apply to latest trunk, it is on top of commit: 869393643de23dcb010cc33091c8eb398de0fd6c
(around Aug 17).

Please feel free to let me know your questions / comments. If everyone agrees with the general
approach, I will go ahead to create a new branch and split the patch.

+ [~jianhe], [~kkaranasos], [~kasha], [~subru], [~curino].

> [Umbrella] Move YARN scheduler towards global scheduler
> -------------------------------------------------------
>                 Key: YARN-5139
>                 URL: https://issues.apache.org/jira/browse/YARN-5139
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: YARN-5139-Global-Schedulingd-esign-and-implementation-notes-v2.pdf,
YARN-5139-Global-Schedulingd-esign-and-implementation-notes.pdf, wip-1.YARN-5139.patch, wip-2.YARN-5139.patch,
wip-3.YARN-5139.patch, wip-4.YARN-5139.patch
> Existing YARN scheduler is based on node heartbeat. This can lead to sub-optimal decisions
because scheduler can only look at one node at the time when scheduling resources.
> Pseudo code of existing scheduling logic looks like:
> {code}
> for node in allNodes:
>    Go to parentQueue
>       Go to leafQueue
>         for application in leafQueue.applications:
>            for resource-request in application.resource-requests
>               try to schedule on node
> {code}
> Considering future complex resource placement requirements, such as node constraints
(give me "a && b || c") or anti-affinity (do not allocate HBase regionsevers and Storm
workers on the same host), we may need to consider moving YARN scheduler towards global scheduling.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message