hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nathan Roberts (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3298) User-limit should be enforced in CapacityScheduler
Date Fri, 06 Mar 2015 17:40:39 GMT

    [ https://issues.apache.org/jira/browse/YARN-3298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14350614#comment-14350614
] 

Nathan Roberts commented on YARN-3298:
--------------------------------------

Hi Wangda. I'm a little concerned about this proposal. I think userlimit has been acting this
way for a long time so a change could have a very significant impact on how queues behave.
 If I'm understanding the proposal correctly, a queue that is configured with minimum_user_limit_percent=50,
capacity=x, max_capacity=10x, user_limit_factor=5 ---- 2 active users would not be able to
get the queue above x. Please correct me if that's not the case. Assuming that is the case,
I'm not sure that's what we want.

What I see user limit doing is controlling which of the actively requesting applications are
getting newly available resources. Basically, making it so that the queue can grow to 10x
in the above example while trying to make sure that each of the users within the queue are
getting equal shares of capacity.   

> User-limit should be enforced in CapacityScheduler
> --------------------------------------------------
>
>                 Key: YARN-3298
>                 URL: https://issues.apache.org/jira/browse/YARN-3298
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacityscheduler, yarn
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>
> User-limit is not treat as a hard-limit for now, it will not consider required-resource
(resource of being-allocated resource request). And also, when user's used resource equals
to user-limit, it will still continue. This will generate jitter issues when we have YARN-2069
(preemption policy kills a container under an user, and scheduler allocate a container under
the same user soon after).
> The expected behavior should be as same as queue's capacity:
> Only when user.usage + required <= user-limit, queue will continue to allocate container.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message