hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nathan Roberts (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3945) maxApplicationsPerUser is wrongly calculated
Date Thu, 30 Jul 2015 22:34:05 GMT

    [ https://issues.apache.org/jira/browse/YARN-3945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14648418#comment-14648418

Nathan Roberts commented on YARN-3945:

Hi [~leftnoteasy]. Regarding minimum_user_limit_percent. 
- I totally agree it is very confusing. 
- I don't think we can change it in any significant way at this point without a major configuration
switch that clearly indicates you're getting different behavior. I'm sure admins have built
up clusters with this tuned in very specific ways, a significant change wouldn't be compatible
with their expectations.

bq. User-limit is not a fairness mechanism to balance resources between users, instead, it
can lead to bad imbalance. One example is, if we set user-limit = 50, and there are 10 users
running, we cannot manage how much resource can be used by each user.
I don't really agree with this. It may not be doing an ideal job but I think the intent is
to introduce fairness between users. It's a progression from 0 being the most fair, and 100+
being more fifo. In your example it's trying to get everyone 50% which isn't likely to happen
so in this case it's going to operate mostly fifo. If the intent is to be much more fair across
the 10 users, then a much smaller value would be appropriate. 

bq. meaningful since #active-user is changing every minute, it is not a predictable formula.
Since the scheduler can't predict what an application is going to request in the future, I
don't see how a predictable formula is even possible (ignoring the possibility of taking away
resources via in-queue preemption). It's not great, but being fair to currently requesting
users makes some bit of sense. 

bq. Instead we may need to consider some notion like fair sharing: user-limit-factor becomes
max-resource-limit of each user, and user-limit-percentage becomes something like guaranteed-concurrent-#user,
when #user > guaranteed-concurrent-#user, rest users can only get idle shares.
user-limit-factor is the max-resource-limit of each user today, right? The second one seems
very hard to track. It seems like one of the initial users can stay in the "guaranteed" set
as long as he keeps requesting resources. This doesn't seem very fair to the users only getting
idle shares. 

> maxApplicationsPerUser is wrongly calculated
> --------------------------------------------
>                 Key: YARN-3945
>                 URL: https://issues.apache.org/jira/browse/YARN-3945
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacityscheduler
>    Affects Versions: 2.7.1
>            Reporter: Naganarasimha G R
>            Assignee: Naganarasimha G R
>         Attachments: YARN-3945.20150728-1.patch, YARN-3945.20150729-1.patch
> maxApplicationsPerUser is currently calculated based on the formula
> {{maxApplicationsPerUser = (int)(maxApplications * (userLimit / 100.0f) * userLimitFactor)}}
but description of userlimit is 
> {quote}
> Each queue enforces a limit on the percentage of resources allocated to a user at any
given time, if there is demand for resources. The user limit can vary between a minimum and
maximum value.{color:red} The the former (the minimum value) is set to this property value
{color} and the latter (the maximum value) depends on the number of users who have submitted
applications. For e.g., suppose the value of this property is 25. If two users have submitted
applications to a queue, no single user can use more than 50% of the queue resources. If a
third user submits an application, no single user can use more than 33% of the queue resources.
With 4 or more users, no user can use more than 25% of the queues resources. A value of 100
implies no user limits are imposed. The default is 100. Value is specified as a integer.
> {quote}
> configuration related to minimum limit should not be made used in a formula to calculate
max applications for a user

This message was sent by Atlassian JIRA

View raw message