hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Payne (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5892) Capacity Scheduler: Support user-specific minimum user limit percent
Date Thu, 25 May 2017 21:32:04 GMT

    [ https://issues.apache.org/jira/browse/YARN-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16025425#comment-16025425

Eric Payne commented on YARN-5892:

Thank you for the reviews.
bq. allUsersTimesWeights will be less than 1. I think in this case UL value is higher

[~sunilg], I think this is similar to the question I answered [above|https://issues.apache.org/jira/browse/YARN-5892?focusedCommentId=15972782&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15972782],
but I'll restate it here for the sake of clarity.

Having a combined sum of weights < 1 does not cause UL to be too large. This is because
{{userLimitResource}} (the return value of {{computeUserLimit}}) is only ever used by {{getComputedResourceLimitFor\[Active|All\]Users}},
which then multiplies the value of {{userLimitResource}} by the appropriate user's weight
before returning it. This will result in the correct value of userLimit for each specific
user. When the sum of active user(s)'s weight(s) is < 1, then it is true that {{userLimitResource}}
is larger than the actual number of resources used. However, {{userLimitResource}} is just
an intermediate value.

bq. In UserManager, do we also need to lock while updating "activeUsersTimesWeights". 
Can you please clarify where you see it being read or written outside the lock? I think the
code is within locks everwhere it is used.

bq. 1) Could you move CapacitySchedulerQueueManager#updateUserWeights to LeafQueue#setupQueueConfigs.

[~leftnoteasy], good optimization. I will make this change, do testing and await [~sunilg]'s
response before submitting a new patch.

> Capacity Scheduler: Support user-specific minimum user limit percent
> --------------------------------------------------------------------
>                 Key: YARN-5892
>                 URL: https://issues.apache.org/jira/browse/YARN-5892
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: capacityscheduler
>            Reporter: Eric Payne
>            Assignee: Eric Payne
>         Attachments: Active users highlighted.jpg, YARN-5892.001.patch, YARN-5892.002.patch,
YARN-5892.003.patch, YARN-5892.004.patch, YARN-5892.005.patch, YARN-5892.006.patch, YARN-5892.007.patch,
YARN-5892.008.patch, YARN-5892.009.patch, YARN-5892.010.patch, YARN-5892.012.patch, YARN-5892.013.patch
> Currently, in the capacity scheduler, the {{minimum-user-limit-percent}} property is
per queue. A cluster admin should be able to set the minimum user limit percent on a per-user
basis within the queue.
> This functionality is needed so that when intra-queue preemption is enabled (YARN-4945
/ YARN-2113), some users can be deemed as more important than other users, and resources from
VIP users won't be as likely to be preempted.
> For example, if the {{getstuffdone}} queue has a MULP of 25 percent, but user {{jane}}
is a power user of queue {{getstuffdone}} and needs to be guaranteed 75 percent, the properties
for {{getstuffdone}} and {{jane}} would look like this:
> {code}
>   <property>
>     <name>yarn.scheduler.capacity.root.getstuffdone.minimum-user-limit-percent</name>
>     <value>25</value>
>   </property>
>   <property>
>     <name>yarn.scheduler.capacity.root.getstuffdone.jane.minimum-user-limit-percent</name>
>     <value>75</value>
>   </property>
> {code}

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message