hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
Date Tue, 24 May 2016 21:47:12 GMT

     [ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Wangda Tan updated YARN-5139:
    Attachment: wip-1.YARN-5139.patch

Uploaded a preliminary and WIP POC patch for Capacity Scheduler changes to help you better
understanding my proposal. The patch includes a {{NodeCandidates}} class, which will be constructed
by scheduler before invokin rootQueue.assignContainers(...).

Then ParentQueue/LeafQueue pass the {{NodeCandidates}} down to application. And application
can sort nodes (using implementation of {{SchedulerNodesScorer}}) and allocate resources.

This idea is very close to approach mentioned in Google Borg paper. Except Google Borg select
requests based on quota/priority and we select requests based on hierarchy of queues/applications.

This patch cannot be compiled, I will try to finish the POC patch with few more examples like
global scheduling filter/scorer for locality, anti-affinity, etc. while waiting for your feedbacks.

> [Umbrella] Move YARN scheduler towards global scheduler
> -------------------------------------------------------
>                 Key: YARN-5139
>                 URL: https://issues.apache.org/jira/browse/YARN-5139
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: wip-1.YARN-5139.patch
> Existing YARN scheduler is based on node heartbeat. This can lead to sub-optimal decisions
because scheduler can only look at one node at the time when scheduling resources.
> Pseudo code of existing scheduling logic looks like:
> {code}
> for node in allNodes:
>    Go to parentQueue
>       Go to leafQueue
>         for application in leafQueue.applications:
>            for resource-request in application.resource-requests
>               try to schedule on node
> {code}
> Considering future complex resource placement requirements, such as node constraints
(give me "a && b || c") or anti-affinity (do not allocate HBase regionsevers and Storm
workers on the same host), we may need to consider moving YARN scheduler towards global scheduling.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message