hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3945) maxApplicationsPerUser is wrongly calculated
Date Tue, 12 Jan 2016 03:59:39 GMT

    [ https://issues.apache.org/jira/browse/YARN-3945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15093246#comment-15093246

Wangda Tan commented on YARN-3945:

Hi [~Naganarasimha],

I agree that fixing user limit is a non-trivial fix and should be done in a separated JIRA.

Thinking hard of this issue, I feel maybe we shouldn't bother user-limit to max-application
too much:
- We have max-am-percent already, and CS respects it
- With above, maybe we don't have to limit number of applications per user, I felt it's not
that important.
- Two different dimensions of limitation (max-am-resource-per-user and max-number-per-user)
could lead to under utilization. (A user could use less AM resource but cannot get new AM
container allocated before app-number exceeds.

I would suggest to make a simple fix:
Now numAppsPerUser could be more than numAppsPerQueue (before of user-limit).
Same to user-resource and user-am-resource, it will be helpful to make sure they're capped
by queue's limitation (am-resource, number-am, queue-max-resource, etc.).

With this user will not confused by web UI reports max resource of a user could exceed max
resource of a queue.

> maxApplicationsPerUser is wrongly calculated
> --------------------------------------------
>                 Key: YARN-3945
>                 URL: https://issues.apache.org/jira/browse/YARN-3945
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacityscheduler
>    Affects Versions: 2.7.1
>            Reporter: Naganarasimha G R
>            Assignee: Naganarasimha G R
>         Attachments: YARN-3945.20150728-1.patch, YARN-3945.20150729-1.patch, YARN-3945.V1.003.patch
> maxApplicationsPerUser is currently calculated based on the formula
> {{maxApplicationsPerUser = (int)(maxApplications * (userLimit / 100.0f) * userLimitFactor)}}
but description of userlimit is 
> {quote}
> Each queue enforces a limit on the percentage of resources allocated to a user at any
given time, if there is demand for resources. The user limit can vary between a minimum and
maximum value.{color:red} The the former (the minimum value) is set to this property value
{color} and the latter (the maximum value) depends on the number of users who have submitted
applications. For e.g., suppose the value of this property is 25. If two users have submitted
applications to a queue, no single user can use more than 50% of the queue resources. If a
third user submits an application, no single user can use more than 33% of the queue resources.
With 4 or more users, no user can use more than 25% of the queues resources. A value of 100
implies no user limits are imposed. The default is 100. Value is specified as a integer.
> {quote}
> configuration related to minimum limit should not be made used in a formula to calculate
max applications for a user

This message was sent by Atlassian JIRA

View raw message