hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
Date Fri, 01 Dec 2017 18:17:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16274720#comment-16274720

Wangda Tan commented on YARN-5139:

Thanks [~Tao Yang] for reporting this. I'm glad to see this runs in your prod environment
for half a year!  

Yeah, please share the use cases of multiple nodes look up from your POV. We can incorporate
it once working on implementations.

In terms of the future work, despite bug fixes, they will be:
1) Global Scheduler Related Refactorings:
(YARN-7438). Additional changes to make SchedulingPlacementSet agnostic to ResourceRequest
/ placement algorithm.
Wangda Tan
(YARN-7457). Delay scheduling should be an individual policy instead of part of scheduler
(YARN-7496). Add muti node lookup support for better placement.

The main purpose is to make a more self-contained per-app placement allocator algorithms.
We're trying to close YARN-7438 soon, and [~sunilg] is working on YARN-7496. If you have interests/bandwidth,
you may take a crack at YARN-7457, which is also crucial to make a clean separation of allocation

2) Additional placement algorithms: 

They're all located under YARN-6592 branch. Currently, most API patches are get committed.
And we're trying to finish simple intra-app affinity/anti-affinity feature as the first use
case of the scheduling constraint in the short term. [~asuresh]/[~kkaranasos] are working
on more advanced allocation algorithm which can aggregate requests from different apps/services
and run LP solver to better place services with picky scheduling constraints. 

Please let me know your suggestions and welcome to participate this work if you have interest!

> [Umbrella] Move YARN scheduler towards global scheduler
> -------------------------------------------------------
>                 Key: YARN-5139
>                 URL: https://issues.apache.org/jira/browse/YARN-5139
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: Explanantions of Global Scheduling (YARN-5139) Implementation.pdf,
YARN-5139-Concurrent-scheduling-performance-report.pdf, YARN-5139-Global-Schedulingd-esign-and-implementation-notes-v2.pdf,
YARN-5139-Global-Schedulingd-esign-and-implementation-notes.pdf, YARN-5139.000.patch, wip-1.YARN-5139.patch,
wip-2.YARN-5139.patch, wip-3.YARN-5139.patch, wip-4.YARN-5139.patch, wip-5.YARN-5139.patch
> Existing YARN scheduler is based on node heartbeat. This can lead to sub-optimal decisions
because scheduler can only look at one node at the time when scheduling resources.
> Pseudo code of existing scheduling logic looks like:
> {code}
> for node in allNodes:
>    Go to parentQueue
>       Go to leafQueue
>         for application in leafQueue.applications:
>            for resource-request in application.resource-requests
>               try to schedule on node
> {code}
> Considering future complex resource placement requirements, such as node constraints
(give me "a && b || c") or anti-affinity (do not allocate HBase regionsevers and Storm
workers on the same host), we may need to consider moving YARN scheduler towards global scheduling.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message