hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3298) User-limit should be enforced in CapacityScheduler
Date Sat, 07 Mar 2015 00:04:39 GMT

    [ https://issues.apache.org/jira/browse/YARN-3298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351171#comment-14351171

Wangda Tan commented on YARN-3298:

Hi [~nroberts],
In your example, each user can still get 5x (after my proposal).

*According to how user-limit get computed:* (my proposal doesn't change this part) 
current-capacity = queue.used + now-required (assume queue's usage is more than queue's capacity)
user-limit = min(max(current-capacity / #active-users, current-capacity * user-limit / 100),
queue-capacity * user-limit-factor)

I realized maybe you mis-understood "user-limit" to be "user-limit" option only, but actually
what I meant is the above equation :).

bq. What I see user limit doing is controlling which of the actively requesting applications
are getting newly available resources. Basically, making it so that the queue can grow to
10x in the above example while trying to make sure that each of the users within the queue
are getting equal shares of capacity.
This will be enforced, each user will allow to use more than queue's minimum share, and can
grow up get equal share of capacity when user-limit and user-limit-factor is properly set.

The only difference is, in the past, each user can get (5x + 1 container resource), but after
this patch, each user can get <= 5x resource.

Does this make sense to you?

> User-limit should be enforced in CapacityScheduler
> --------------------------------------------------
>                 Key: YARN-3298
>                 URL: https://issues.apache.org/jira/browse/YARN-3298
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacityscheduler, yarn
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
> User-limit is not treat as a hard-limit for now, it will not consider required-resource
(resource of being-allocated resource request). And also, when user's used resource equals
to user-limit, it will still continue. This will generate jitter issues when we have YARN-2069
(preemption policy kills a container under an user, and scheduler allocate a container under
the same user soon after).
> The expected behavior should be as same as queue's capacity:
> Only when user.usage + required <= user-limit, queue will continue to allocate container.

This message was sent by Atlassian JIRA

View raw message