hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5892) Capacity Scheduler: Support user-specific minimum user limit percent
Date Fri, 31 Mar 2017 19:30:42 GMT

    [ https://issues.apache.org/jira/browse/YARN-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951539#comment-15951539

Wangda Tan commented on YARN-5892:

Thanks [~eepayne] to update the patch.

Several comments:

1) For User weight:
- I prefer to only allow specify it in the leaf queue level. Inheriting from parent causes
confusing in other cases. (For example should we allow overwriting in leaf, etc.)
- I think we should limit the user weight to be: {{User_weight <= 100 / MULP}} when validating
the config. Otherwise if MULP = 100, and user X's weight = 3, by definition it can get guaranteed
resource of 3 * queue configured resource, which is not good.

2) LeafQueue: 
- getUserAMResourceLimitPerPartition(String) is not used by anyone, can be removed.

3) UsersManager:
I think logics in the {{computeUserLimit}} should be updated:
    int usersCount = getNumActiveUsers();
    Resource resourceUsed = totalResUsageForActiveUsers.getUsed(nodePartition);

    // For non-activeUser calculation, consider all users count.
    if (!activeUser) {
      resourceUsed = currentCapacity;
      usersCount = users.size();
Users_count should be a float and impacted by weight according to definition of weight (weight=x
means equivalent to x users). For example, if a two users in the queue, one has weight=2,
one has weight = 0.5, the users count should be 2.5. And we need get separately users_count_by_weight
for all users and active users.

4) updateUsersWeights:
Should we move userLimitNeedsRecompute into the weight lock, otherwise it could cause a race

5) Could you update doc with more detailed example?

> Capacity Scheduler: Support user-specific minimum user limit percent
> --------------------------------------------------------------------
>                 Key: YARN-5892
>                 URL: https://issues.apache.org/jira/browse/YARN-5892
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: capacityscheduler
>            Reporter: Eric Payne
>            Assignee: Eric Payne
>         Attachments: Active users highlighted.jpg, YARN-5892.001.patch, YARN-5892.002.patch,
YARN-5892.003.patch, YARN-5892.004.patch, YARN-5892.005.patch, YARN-5892.006.patch
> Currently, in the capacity scheduler, the {{minimum-user-limit-percent}} property is
per queue. A cluster admin should be able to set the minimum user limit percent on a per-user
basis within the queue.
> This functionality is needed so that when intra-queue preemption is enabled (YARN-4945
/ YARN-2113), some users can be deemed as more important than other users, and resources from
VIP users won't be as likely to be preempted.
> For example, if the {{getstuffdone}} queue has a MULP of 25 percent, but user {{jane}}
is a power user of queue {{getstuffdone}} and needs to be guaranteed 75 percent, the properties
for {{getstuffdone}} and {{jane}} would look like this:
> {code}
>   <property>
>     <name>yarn.scheduler.capacity.root.getstuffdone.minimum-user-limit-percent</name>
>     <value>25</value>
>   </property>
>   <property>
>     <name>yarn.scheduler.capacity.root.getstuffdone.jane.minimum-user-limit-percent</name>
>     <value>75</value>
>   </property>
> {code}

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message