hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Payne (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4108) CapacityScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request
Date Wed, 16 Mar 2016 00:35:33 GMT

    [ https://issues.apache.org/jira/browse/YARN-4108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15196551#comment-15196551
] 

Eric Payne commented on YARN-4108:
----------------------------------

Thanks, [~leftnoteasy], for all of the great effort on this JIRA. This feature is definitely
needed.

 I do have one concern that I feel still needs to be addressed.

The use case is as follows:
|| name || guaranteed || used || pending ||
| queueA | 33 | 45 | 5 |
| queueB | 33 | 55 | 0 |
| queueC | 34 |  0 | 0 |

Prior to YARN-4108, the scheduler will preempt 5 resources from {{queueB}} and give them to
{{queueA}}, even though both {{queueB}} and {{queueA}} are over their guaranteed resources.
Does the code in LeafQueue#setPreemptionAllowed change that behavior? I see the following:
{code}
    limits.setIsAllowPreemption(usedCapacity < guaranteedCapacity);
{code}
In the above use case, would this code prevent {{queueA}} from getting the 5 resources because
{{queueA}}'s used is above its guaranteed? [This was discussed some above|https://issues.apache.org/jira/browse/YARN-4108?focusedCommentId=15101166&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15101166],
but I wanted to make sure that, if this behavior is changed, we are all in agreement that
this is the best approach.

> CapacityScheduler: Improve preemption to preempt only those containers that would satisfy
the incoming request
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-4108
>                 URL: https://issues.apache.org/jira/browse/YARN-4108
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacity scheduler
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: YARN-4108-design-doc-V3.pdf, YARN-4108-design-doc-v1.pdf, YARN-4108-design-doc-v2.pdf,
YARN-4108.1.patch, YARN-4108.10.patch, YARN-4108.11.patch, YARN-4108.2.patch, YARN-4108.3.patch,
YARN-4108.4.patch, YARN-4108.5.patch, YARN-4108.6.patch, YARN-4108.7.patch, YARN-4108.8.patch,
YARN-4108.9.patch, YARN-4108.poc.1.patch, YARN-4108.poc.2-WIP.patch, YARN-4108.poc.3-WIP.patch,
YARN-4108.poc.4-WIP.patch
>
>
> This is sibling JIRA for YARN-2154. We should make sure container preemption is more
effective.
> *Requirements:*:
> 1) Can handle case of user-limit preemption
> 2) Can handle case of resource placement requirements, such as: hard-locality (I only
want to use rack-1) / node-constraints (YARN-3409) / black-list (I don't want to use rack1
and host\[1-3\])
> 3) Can handle preemption within a queue: cross user preemption (YARN-2113), cross applicaiton
preemption (such as priority-based (YARN-1963) / fairness-based (YARN-3319)).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message