hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2056) Disable preemption at Queue level
Date Fri, 19 Sep 2014 10:33:34 GMT

    [ https://issues.apache.org/jira/browse/YARN-2056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14140295#comment-14140295
] 

Wangda Tan commented on YARN-2056:
----------------------------------

Hi [~eepayne],
Thanks for updating, I've took a look at your patch and completely re-thought about it. Some
ideas:

In existing preemption logic, the {{resetCapacity}} is a static calculation, irrelatated to
resource usage of any queue.
And with result calculated by {{resetCapacity}}, {{computeFixpointAllocation}} will make sure
that: *If there's any resource isn't allocated, it will be assigned to existing queues by
its normalized_guarantee calculated*. This is foundation of following calculation, like get
{{toBePreempted}} of each queue, we shouldn't break it.

IMHO, the right place to put reserving resource logic for un-preemptable queue is not {{resetCapacity}},
it should in {{computeFixpointAllocation}}.

I think adding {{preemptableExtra}} in each TempQueue, and ParentQueue will accumulate {{preemptableExtra}}
of its children is a good idea.

What I proposed to do in {{computeFixpointAllocation}}

{code}
computeFixpointAllocation {
    ...

+	for (q in qAlloc) {
+		if (q.disablePreempt) {
+			q.ideal_assigned = q.current
+           unassigned = unassigned - q.ideal_assigned
+		}
+	}

    while (!qAlloc.isEmpty()
        && Resources.greaterThan(rc, tot_guarant, unassigned, Resources.none())
        && Resources.greaterThan(rc, tot_guarant, wQassigned, Resources.none())) {
        ...
    }
}
{code}

And in the while (!qAlloc.isEmpty()) loop above, we need take care the fact: some queue's
ideal_assigned is started from 0, and some queue's ideal_assigned is started from current.

Does this make sense to you?

Wangda

> Disable preemption at Queue level
> ---------------------------------
>
>                 Key: YARN-2056
>                 URL: https://issues.apache.org/jira/browse/YARN-2056
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>    Affects Versions: 2.4.0
>            Reporter: Mayank Bansal
>            Assignee: Eric Payne
>         Attachments: YARN-2056.201408202039.txt, YARN-2056.201408260128.txt, YARN-2056.201408310117.txt,
YARN-2056.201409022208.txt, YARN-2056.201409181916.txt
>
>
> We need to be able to disable preemption at individual queue level



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message