hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wei Shao (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-3807) Proposal of Guaranteed Capacity Scheduling for YARN
Date Tue, 16 Jun 2015 16:06:05 GMT

     [ https://issues.apache.org/jira/browse/YARN-3807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Wei Shao updated YARN-3807:
---------------------------
    Description: 
This proposal talks about limitations of the YARN scheduling policies for SLA applications,
and tries to solve them by YARN-3806 and the new scheduling policy called guaranteed capacity
scheduling.
Guaranteed capacity scheduling makes guarantee to the applications that they can get resources
under specified capacity cap in totally predictable manner. The application can meet SLA more
easily since it is self-contained in the shared cluster - external uncertainties are eliminated.
For example, suppose queue A has initial capacity 100G memory, and there are two pending applications
1 and 2, 1’s specified capacity is 70G, 2’s specified capacity is 50G. Queue A may accept
application 1 to run first and makes guarantee that 1 can get resources exponentially up to
its capacity and won’t be preempted (if allocation of 1 is 5G in scheduling cycle N, demand
is 80G, exponential factor is 2. In N+1, it can get 5G, in N+2, it can get 10G, in N+3, it
can get 20G, and in N+4, it can get 30G, reach its capacity). Later, when the cluster is free,
queue A may decide to scale up by increasing its capacity to 120G, so it can accept application
2 and make guarantee to it as well. Queue A can scale down to its initial capacity when any
application completes.
Guaranteed capacity scheduling also have some other features. See proposal for more details.

  was:
This proposal talks about limitations of the YARN scheduling policies for SLA applications,
and tries to solve them by YARN-3806 and the new scheduling policy called guaranteed capacity
scheduling.
Guaranteed capacity scheduling makes guarantee to the applications that they can get resources
under specified capacity cap in totally predictable manner. The application can meet SLA more
easily since it is self-contained in the shared cluster - external uncertainties are eliminated.


> Proposal of Guaranteed Capacity Scheduling for YARN
> ---------------------------------------------------
>
>                 Key: YARN-3807
>                 URL: https://issues.apache.org/jira/browse/YARN-3807
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: capacityscheduler, fairscheduler
>            Reporter: Wei Shao
>         Attachments: ProposalOfGuaranteedCapacitySchedulingForYARN-V1.0.pdf
>
>
> This proposal talks about limitations of the YARN scheduling policies for SLA applications,
and tries to solve them by YARN-3806 and the new scheduling policy called guaranteed capacity
scheduling.
> Guaranteed capacity scheduling makes guarantee to the applications that they can get
resources under specified capacity cap in totally predictable manner. The application can
meet SLA more easily since it is self-contained in the shared cluster - external uncertainties
are eliminated.
> For example, suppose queue A has initial capacity 100G memory, and there are two pending
applications 1 and 2, 1’s specified capacity is 70G, 2’s specified capacity is 50G. Queue
A may accept application 1 to run first and makes guarantee that 1 can get resources exponentially
up to its capacity and won’t be preempted (if allocation of 1 is 5G in scheduling cycle
N, demand is 80G, exponential factor is 2. In N+1, it can get 5G, in N+2, it can get 10G,
in N+3, it can get 20G, and in N+4, it can get 30G, reach its capacity). Later, when the cluster
is free, queue A may decide to scale up by increasing its capacity to 120G, so it can accept
application 2 and make guarantee to it as well. Queue A can scale down to its initial capacity
when any application completes.
> Guaranteed capacity scheduling also have some other features. See proposal for more details.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message