hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Payne (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-2056) Disable preemption at Queue level
Date Tue, 23 Sep 2014 23:55:35 GMT

     [ https://issues.apache.org/jira/browse/YARN-2056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Eric Payne updated YARN-2056:
    Attachment: YARN-2056.201409232329.txt


This patch now works in all cases, except IF queue preemption is disabled for a particular
queue, it has a different behavior than previously. That is:
root has A,B,C, total capacity = 90
A.guaranteed = 30, A.pending = 20, A.current = 40
B.guaranteed = 30, B.pending = 0, B.current = 50
C.guaranteed = 30, C.pending = 0, C.current = 0
In the above case, if all queues are preemptable, this patch works the same as it did before.
That is, A and B will both end up with 45.

However, with this patch
IF (A is not preemptable)
    AND (A is already over capacity)
    AND (all resources are used)
    AND (A is asking for more resources) {
        A will remain at 40 and B will remain at 50
I believe that there is a way to make the this patch maintain the old behavior, even if A
is not preemptable, but it would require something like the following algorithm:
In ProportionalCapacityPreemptionPolicy#computeFixpointAllocation:
FOR each queue {
    IF queue has untouchableExtra {
        queue.idalAssigned = queue.guaranteed + queue.untouchable
        unassigned -= queue.idealAssigned
        remove queue from qAlloc
        add queue to list of queues that were removed

Assign the remaining unassigned resources, computing idealAssigned for the remaining queues,
with the following modification:
    IF (queues at this level go over their capacity)
        AND (they are over by the same percentage as the queue(s) that were removed) {
        put the removed queue(s) back into qAlloc and continue to compute idealAssigned

> Disable preemption at Queue level
> ---------------------------------
>                 Key: YARN-2056
>                 URL: https://issues.apache.org/jira/browse/YARN-2056
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>    Affects Versions: 2.4.0
>            Reporter: Mayank Bansal
>            Assignee: Eric Payne
>         Attachments: YARN-2056.201408202039.txt, YARN-2056.201408260128.txt, YARN-2056.201408310117.txt,
YARN-2056.201409022208.txt, YARN-2056.201409181916.txt, YARN-2056.201409210049.txt, YARN-2056.201409232329.txt
> We need to be able to disable preemption at individual queue level

This message was sent by Atlassian JIRA

View raw message