hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2009) Priority support for preemption in ProportionalCapacityPreemptionPolicy
Date Mon, 19 Sep 2016 23:10:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-2009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15504978#comment-15504978

Wangda Tan commented on YARN-2009:


bq. ... then I don't think idealAssigned can be calculated independently from each other ...

Actually I was thinking the same thing which we should compute idealAssigned for each user
when I was reviewing YARN-2069. But I realized we may not need, let me explain it a little
Computed user-limit resource in existing CS is using as higher bound of how much that each
user should get, there's no "lower bound user limit resource" in reality. 

I think all of us agree that behavior of preemption should be consistent with behavior of
scheduling, any mismatch between the two could lead to excessive preemption.

When FIFO (and also FIFO + PRIORITY) policy is enabled, an example of existing CS's behavior
Queue's user-limit-percent = 33
Queue's used=guaranteed=max=12. 
There're 3 users (A,B,C) in the queue, order of applications are A/B/C
Applications from user-A/C are asking for more resource, and application of userB is satisfied

So the computed user-limit-resource will be 6.

Assume resource usages of A/B/C are 5/6/1, and A/C have 1 pending resource, 

The actual user-ideal-assignment when doing scheduling is 6/6/0 !
(A can get the 1 additional resource, and B will not changed, C can get nothing after that)

So in another words, user-limit is just a cap in additional to FIFO (or FIFO+Priority) order:

Back to the preemption patch, the pseudo code to compute application ideal allocation consider
user limit will be:
void compute-ideal-allocation-for-apps(List<Application> apps) {
    user-limit-resource = queue.get-user-limit-resource();

    // initial all value to 0
    Map<String, Resource> user-to-allocated;

    for app in sort-by-fifo-or-priority(apps) {
       if (user-to-allocated.get(app.user) < user-limit-resource) {
            app.allocated = min(app.used + pending, user-limit-resource - user-to-allocated.get(app.user));
            user-to-allocated.get(app.user) += app.allocated;
       } else {
             // skip this app because user-limit reached

Please let me know about your thoughts.


> Priority support for preemption in ProportionalCapacityPreemptionPolicy
> -----------------------------------------------------------------------
>                 Key: YARN-2009
>                 URL: https://issues.apache.org/jira/browse/YARN-2009
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacityscheduler
>            Reporter: Devaraj K
>            Assignee: Sunil G
>         Attachments: YARN-2009.0001.patch
> While preempting containers based on the queue ideal assignment, we may need to consider
preempting the low priority application containers first.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message