hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wei Shao (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-3807) Proposal of Guaranteed Capacity Scheduling for YARN
Date Thu, 18 Jun 2015 22:59:01 GMT

     [ https://issues.apache.org/jira/browse/YARN-3807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Wei Shao updated YARN-3807:
    Attachment: ProposalOfGuaranteedCapacitySchedulingForYARN-V1.2.pdf

> Proposal of Guaranteed Capacity Scheduling for YARN
> ---------------------------------------------------
>                 Key: YARN-3807
>                 URL: https://issues.apache.org/jira/browse/YARN-3807
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: capacityscheduler, fairscheduler
>            Reporter: Wei Shao
>         Attachments: ProposalOfGuaranteedCapacitySchedulingForYARN-V1.0.pdf, ProposalOfGuaranteedCapacitySchedulingForYARN-V1.1.pdf,
> This proposal talks about limitations of the YARN scheduling policies for SLA applications,
and tries to solve them by YARN-3806 and the new scheduling policy called guaranteed capacity
> Guaranteed capacity scheduling makes guarantee to the applications that they can get
resources under specified capacity cap in totally predictable manner. The application can
meet SLA more easily since it is self-contained in the shared cluster - external uncertainties
are eliminated.
> For example, suppose queue A has initial capacity 100G memory, and there are two pending
applications 1 and 2, 1’s specified capacity is 70G, 2’s specified capacity is 50G. Queue
A may accept application 1 to run first and makes guarantee that 1 can get resources exponentially
up to its capacity and won’t be preempted (if allocation of 1 is 5G in scheduling cycle
N, demand is 80G, exponential factor is 2. In N+1, it can get 5G, in N+2, it can get 10G,
in N+3, it can get 20G, and in N+4, it can get 30G, reach its capacity). Later, when the cluster
is free, queue A may decide to scale up by increasing its capacity to 120G, so it can accept
application 2 and make guarantee to it as well. Queue A can scale down to its initial capacity
when any application completes.
> Guaranteed capacity scheduling also has other features that the example doesn’t illustrate.
See proposal for more details.

This message was sent by Atlassian JIRA

View raw message