hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4280) CapacityScheduler reservations may not prevent indefinite postponement on a busy cluster
Date Mon, 02 May 2016 14:23:12 GMT

    [ https://issues.apache.org/jira/browse/YARN-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15266673#comment-15266673
] 

Jason Lowe commented on YARN-4280:
----------------------------------

bq. The problem of allowing one container reserved exceed queue's max capacity is: in a small
cluster, one single large container could mean a large proportion of the cluster. And queue's
maximum capacity will be exceeded a lot.

Agreed, we don't want to let a thousand users in a relatively small queue lock down the entire
cluster with reservations.

I think the algorithm needs to work something like this:
- If a user would normally need to make a reservation and is within its user limits but the
queue absolute capacity would be exceeded then that queue is locked down for further container
allocations until any of the following are true:
-- the container can be allocated
-- the reservation can be placed
-- the user limit is lowered and the reservation would no longer fit within the user limit
-- the container request is cancelled
-- a higher priority allocation comes along that can fit
- If a user would normally need to make a reservation and is within both its user limits and
leaf queue absolute capacity _but_ a parent queue's capacity would be exceeded then _all_
queues within that parent queue's hierarchy are locked down from further allocations until
any of the following are true:
-- the container can be allocated
-- the reservation can be placed
-- the user limit is lowered and the reservation would no longer fit within the user limit
-- the container request is cancelled
-- a higher priority allocation (e.g.: from a more underserved queue within the hierarchy)
comes along that can fit

The leaf queue case seems pretty straightforward -- we simply early-out of the assignment
loop if we need to make a reservation but cannot due to the queue's limits.  Then only higher-priority
allocations will be allowed to be made until we can place that reservation.

The parent queue case is more complicated, but I think it can be accomplished in a similar
manner.  When the parent-queue limit is detected by the leaf queue assignment method, it returns
with a result that indicates to the parent queue(s) that no further allocations can be made
due to parent queue limits, and all queues up to the parent queue that also hit that limit
return early from the assignment due to that condition.  Then only higher-priority allocations
(i.e.: from either a more underserved queue in the hierarchy or higher-priority app within
the leaf queue) could potentially get more containers until the pertinent container request
can be addressed either by reservation or allocation.


> CapacityScheduler reservations may not prevent indefinite postponement on a busy cluster
> ----------------------------------------------------------------------------------------
>
>                 Key: YARN-4280
>                 URL: https://issues.apache.org/jira/browse/YARN-4280
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacity scheduler
>    Affects Versions: 2.6.1, 2.8.0, 2.7.1
>            Reporter: Kuhu Shukla
>            Assignee: Kuhu Shukla
>
> Consider the following scenario:
> There are 2 queues A(25% of the total capacity) and B(75%), both can run at total cluster
capacity. There are 2 applications, appX that runs on Queue A, always asking for 1G containers(non-AM)
and appY runs on Queue B asking for 2 GB containers.
> The user limit is high enough for the application to reach 100% of the cluster resource.

> appX is running at total cluster capacity, full with 1G containers releasing only one
container at a time. appY comes in with a request of 2GB container but only 1 GB is free.
Ideally, since appY is in the underserved queue, it has higher priority and should reserve
for its 2 GB request. Since this request puts the alloc+reserve above total capacity of the
cluster, reservation is not made. appX comes in with a 1GB request and since 1GB is still
available, the request is allocated. 
> This can continue indefinitely causing priority inversion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message