Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 1 Dec 2017 09:03:01 +0000 (UTC)
From: "Tao Yang (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12972528.1464126299000.363548.1512118981927@Atlassian.JIRA>
In-Reply-To: <JIRA.12972528.1464126299000@Atlassian.JIRA>
References: <JIRA.12972528.1464126299000@Atlassian.JIRA> <JIRA.12972528.1464126299901@jira-lw-us.apache.org>
Subject: [jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler
 towards global scheduler
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 01 Dec 2017 09:03:11 -0000


    [ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16274131#comment-16274131 ] 

Tao Yang commented on YARN-5139:
--------------------------------

Async-scheduling decouples scheduling from nm heartbeats and use multiple allocation threads to make scheduling more efficient. It's very important for us and we have enabled async-scheduling mode of global scheduler on our production cluster which has thousands of nodes for half a year. All the problems we met were submitted to community and most of them were already resolved.
Glad to hear that this feature are moving on and will pay more attention to multiple nodes lookup (YARN-7494) and node scorer (in design) for better placement, And we would like to participate in this work if needed.

> [Umbrella] Move YARN scheduler towards global scheduler
> -------------------------------------------------------
>
>                 Key: YARN-5139
>                 URL: https://issues.apache.org/jira/browse/YARN-5139
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: Explanantions of Global Scheduling (YARN-5139) Implementation.pdf, YARN-5139-Concurrent-scheduling-performance-report.pdf, YARN-5139-Global-Schedulingd-esign-and-implementation-notes-v2.pdf, YARN-5139-Global-Schedulingd-esign-and-implementation-notes.pdf, YARN-5139.000.patch, wip-1.YARN-5139.patch, wip-2.YARN-5139.patch, wip-3.YARN-5139.patch, wip-4.YARN-5139.patch, wip-5.YARN-5139.patch
>
>
> Existing YARN scheduler is based on node heartbeat. This can lead to sub-optimal decisions because scheduler can only look at one node at the time when scheduling resources.
> Pseudo code of existing scheduling logic looks like:
> {code}
> for node in allNodes:
>    Go to parentQueue
>       Go to leafQueue
>         for application in leafQueue.applications:
>            for resource-request in application.resource-requests
>               try to schedule on node
> {code}
> Considering future complex resource placement requirements, such as node constraints (give me "a && b || c") or anti-affinity (do not allocate HBase regionsevers and Storm workers on the same host), we may need to consider moving YARN scheduler towards global scheduling.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org