hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Payne (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7619) Max AM Resource value in CS UI is different for every user
Date Thu, 07 Dec 2017 23:29:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16282707#comment-16282707

Eric Payne commented on YARN-7619:

All of the other solutions I could think of seem undesirable.

One solution would is to have {{LeafQueue}} remember the last user for which it activated
an application. The resource usages for that user are passed through the {{UserInfo}} object
to {{CapacitySchedulerPage}}, which then extracts the last activated user's AM limit from
those usages. This is not ideal, because it doesn't take into account user weights. So, if
the last activated user has a weight not equal to 1.0, the AM limit may be wrong for some
users. (_On a side note, user weights do not look to be affecting user AM limits even though
{{LeafQueue#getUserAMResourceLimitPerPartition}} seems to be computing the limit using user
weights_). Also, if the last activated user leaves the queue, we have to use each users' AM
limit, which puts us back where we started.

Another solution may be to have {{UsersManager}} sort the users list to be in last-activated-first
order. Then, when {{CapacitySchedulerPage#QueueUsersInfoBlock}} is rendering the users info
block, it could just get the user AM limit from the first uesr. That's what {{CapacitySchedulerPage#LeafQueueInfoBlock}}
does when it's retrieving the value for *Max Application Master Resources Per User*. It just
expects the first one to be the correct one for all the users in the queue.

Ideally, I would say it would be best to save the recomputed user AM limit to all users objects
whenever {{LeafQueue#getUserAMResourceLimitPerPartition}} is called, but that may cause a
significant performance hit. Even so, I think this option is the cleanest and the performance
hit may not be that bad.

> Max AM Resource value in CS UI is different for every user
> ----------------------------------------------------------
>                 Key: YARN-7619
>                 URL: https://issues.apache.org/jira/browse/YARN-7619
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacity scheduler, yarn
>    Affects Versions: 2.9.0, 3.0.0-beta1, 2.8.2, 3.1.0
>            Reporter: Eric Payne
>            Assignee: Eric Payne
>         Attachments: Max AM Resources is Different for Each User.png
> YARN-7245 addressed the problem that the {{Max AM Resource}} in the capacity scheduler
UI used to contain the queue-level AM limit instead of the user-level AM limit. It fixed this
by using the user-specific AM limit that is calculated in {{LeafQueue#activateApplications}},
stored in each user's {{LeafQueue#User}} object, and retrieved via {{UserInfo#getResourceUsageInfo}}.
> The problem is that this user-specific AM limit depends on the activity of other users
and other applications in a queue, and it is only calculated and updated when a user's application
is activated. So, when {{CapacitySchedulerPage}} retrieves the user-specific AM limit, it
is a stale value unless an application was recently activated for a particular user.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message