hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Payne (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5889) Improve user-limit calculation in capacity scheduler
Date Wed, 11 Jan 2017 23:13:17 GMT

    [ https://issues.apache.org/jira/browse/YARN-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15819482#comment-15819482

Eric Payne commented on YARN-5889:

- Shoule {{resetUserAddedOrRemoved}} just be {{setUserAddedOrRemoved}}
- {{LeafQueue}}: I think {{totalUserConsumedRatio}} should be removed, since it'
s not used.
- {{LeafQueue#recalculateULCount}} / {{UsersManager#User#cachedULCount}}: I know I came up
with the name originally, but I think a better name would be {{recalculateUL}}
- {{getComputedActiveUserLimit}} / {{getComputedUserLimit}}: User's {{cachedULCount}} needs
to be updated when the UL is recomputed or else it will always be out of sync and will always
be recomputed:
    if (userLimitPerSchedulingMode == null
        || user.getCachedULCount() != lQueue.getRecalculateULCount()) {
      userLimitPerSchedulingMode = reComputeUserLimits(rc, userName,
          nodePartition, clusterResource, false);

bq. User#setCachedCount, should we invalidateUL for the user who allocates/releases containers,
or we should invalidate all user limit? I think the latter one is more safe to me.
Yes, unfortunately, I think that once the queue goes above its guarantee, the ratio will change
when containers are allocated or released. We may be able to do an optimization to only reset
the specific user's count when the queue is under its guarantee and all users when it is over,
but that may not be worth the added complexity.

> Improve user-limit calculation in capacity scheduler
> ----------------------------------------------------
>                 Key: YARN-5889
>                 URL: https://issues.apache.org/jira/browse/YARN-5889
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacity scheduler
>            Reporter: Sunil G
>            Assignee: Sunil G
>         Attachments: YARN-5889.0001.patch, YARN-5889.0001.suggested.patchnotes, YARN-5889.0002.patch,
YARN-5889.v0.patch, YARN-5889.v1.patch, YARN-5889.v2.patch
> Currently user-limit is computed during every heartbeat allocation cycle with a write
lock. To improve performance, this tickets is focussing on moving user-limit calculation out
of heartbeat allocation flow.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message