hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2009) Priority support for preemption in ProportionalCapacityPreemptionPolicy
Date Fri, 07 Oct 2016 18:37:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-2009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15555895#comment-15555895

Wangda Tan commented on YARN-2009:

bq. I am not comfortable with fixing this in another patch. Our main use case is the one where
multiple users need to use the same queue with apps at the same priority...

[~eepayne], I'm not trying to argue the importance of the use case: {{multiple users need
to use the same queue with apps at the same priority}} -- it's the most important use case
of intra-queue preemption. But to make we can easier review the patch, can we move it to a
separate JIRA and start to work on it right after the JIRA is get committed (if we're generally
fine with existing JIRA). This patch is already very dense, review a patch with changed logics
is much harder than review a separate patch on top of this one.

The major problem of why this patch cannot solve the preemption between user is -- it doesn't
calculate "ideal allocation" and "preemptable resource" for user. Existing logic only caps
calculated ideal allocation by user limit, we need to do more:
For example, when there're two users in a queue, u1 above UL, u2 below UL, however u2 doesn't
have more pending resource, in this case, we shouldn't preempt resource from u1.

To solve the problem, we need add some logics to {{calculateIdealAssignedResourcePerApp}}
to calculate the actual ideal allocation / preemptable for users, and we need update the while
condition in {{validateOutSameAppPriorityFromDemand}} as well. It doesn't sound like a very
simple patch (few lines change) to me.

The change proposal mentioned in your above comment is not correct:
bq. I think it should be something like the following, maybe: ...
This will cause application from the same user preempt according to FIFO order (A1/A2 from
the same user, A1 submitted first, and ask for more resource, A2 can preempt resource from
A2), which is what we want to avoid.

And this doesn't sound correct to me as well:
bq. I think that instead, it should order the apps by how much they are underserved. I think
that it should be ordering the apps by tmpApp.toBePreemptedByOther instead of priority...
It could preempt violate FIFO order, and in the scheduling cycle, preempted resource may come
back to the app which is just get preempted.

Please let me if you have any concerns to move the user limit preemption to a separate JIRA.


> Priority support for preemption in ProportionalCapacityPreemptionPolicy
> -----------------------------------------------------------------------
>                 Key: YARN-2009
>                 URL: https://issues.apache.org/jira/browse/YARN-2009
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacityscheduler
>            Reporter: Devaraj K
>            Assignee: Sunil G
>         Attachments: YARN-2009.0001.patch, YARN-2009.0002.patch, YARN-2009.0003.patch,
YARN-2009.0004.patch, YARN-2009.0005.patch
> While preempting containers based on the queue ideal assignment, we may need to consider
preempting the low priority application containers first.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message