Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 7 Dec 2017 23:29:00 +0000 (UTC)
From: "Eric Payne (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.13123239.1512579006000.413442.1512689340204@Atlassian.JIRA>
In-Reply-To: <JIRA.13123239.1512579006000@Atlassian.JIRA>
References: <JIRA.13123239.1512579006000@Atlassian.JIRA> <JIRA.13123239.1512579006243@jira-lw-us.apache.org>
Subject: [jira] [Commented] (YARN-7619) Max AM Resource value in CS UI is
 different for every user
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 07 Dec 2017 23:29:05 -0000


    [ https://issues.apache.org/jira/browse/YARN-7619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16282707#comment-16282707 ] 

Eric Payne commented on YARN-7619:
----------------------------------

All of the other solutions I could think of seem undesirable.

One solution would is to have {{LeafQueue}} remember the last user for which it activated an application. The resource usages for that user are passed through the {{UserInfo}} object to {{CapacitySchedulerPage}}, which then extracts the last activated user's AM limit from those usages. This is not ideal, because it doesn't take into account user weights. So, if the last activated user has a weight not equal to 1.0, the AM limit may be wrong for some users. (_On a side note, user weights do not look to be affecting user AM limits even though {{LeafQueue#getUserAMResourceLimitPerPartition}} seems to be computing the limit using user weights_). Also, if the last activated user leaves the queue, we have to use each users' AM limit, which puts us back where we started.

Another solution may be to have {{UsersManager}} sort the users list to be in last-activated-first order. Then, when {{CapacitySchedulerPage#QueueUsersInfoBlock}} is rendering the users info block, it could just get the user AM limit from the first uesr. That's what {{CapacitySchedulerPage#LeafQueueInfoBlock}} does when it's retrieving the value for *Max Application Master Resources Per User*. It just expects the first one to be the correct one for all the users in the queue.

Ideally, I would say it would be best to save the recomputed user AM limit to all users objects whenever {{LeafQueue#getUserAMResourceLimitPerPartition}} is called, but that may cause a significant performance hit. Even so, I think this option is the cleanest and the performance hit may not be that bad.

> Max AM Resource value in CS UI is different for every user
> ----------------------------------------------------------
>
>                 Key: YARN-7619
>                 URL: https://issues.apache.org/jira/browse/YARN-7619
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacity scheduler, yarn
>    Affects Versions: 2.9.0, 3.0.0-beta1, 2.8.2, 3.1.0
>            Reporter: Eric Payne
>            Assignee: Eric Payne
>         Attachments: Max AM Resources is Different for Each User.png
>
>
> YARN-7245 addressed the problem that the {{Max AM Resource}} in the capacity scheduler UI used to contain the queue-level AM limit instead of the user-level AM limit. It fixed this by using the user-specific AM limit that is calculated in {{LeafQueue#activateApplications}}, stored in each user's {{LeafQueue#User}} object, and retrieved via {{UserInfo#getResourceUsageInfo}}.
> The problem is that this user-specific AM limit depends on the activity of other users and other applications in a queue, and it is only calculated and updated when a user's application is activated. So, when {{CapacitySchedulerPage}} retrieves the user-specific AM limit, it is a stale value unless an application was recently activated for a particular user.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org