hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5864) Capacity Scheduler preemption for fragmented cluster
Date Thu, 15 Dec 2016 22:29:58 GMT

    [ https://issues.apache.org/jira/browse/YARN-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15752695#comment-15752695

Wangda Tan commented on YARN-5864:

Offline discussed with [~vinodkv].

We can have a better semantic of this feature, which is we can add queue-priority property.
(Credit to [~vinodkv] for the idea).

In existing scheduler, we sort queues based on (used-capacity / configured-capacity). But
in some cases we have some apps/services need get resource first. For example, we allocate
85% to production queue, and 15% to test queue. When production queue is underutilized, we
want scheduler give resource to production queue first regardless of test queue's utilization.

A rough plan is: we will assign priority to queues under the same parent. Each time scheduler
picks underutilized queue with highest priority, if there's no underutilized queue, scheduler
picks queue with lowest utilization.

And when we do preemption, if queue with higher priority has some special resource requests,
such as very large memory, hard locality, placement constraint, etc. Scheduler will do relatively
*conservative* preemption from other queues with lower priority regardless of utilization.

That is just a rough idea, [~curino] please let us know your comments. I can formalize the
design once we can agree with the approach generally.

> Capacity Scheduler preemption for fragmented cluster 
> -----------------------------------------------------
>                 Key: YARN-5864
>                 URL: https://issues.apache.org/jira/browse/YARN-5864
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: YARN-5864.poc-0.patch
> YARN-4390 added preemption for reserved container. However, we found one case that large
container cannot be allocated even if all queues are under their limit.
> For example, we have:
> {code}
> Two queues, a and b, capacity 50:50 
> Two nodes: n1 and n2, each of them have 50 resource 
> Now queue-a uses 10 on n1 and 10 on n2
> queue-b asks for one single container with resource=45. 
> {code} 
> The container could be reserved on any of the host, but no preemption will happen because
all queues are under their limits. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message