hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
Date Mon, 21 Aug 2017 15:56:07 GMT

    [ https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16135351#comment-16135351
] 

ASF GitHub Bot commented on YARN-2162:
--------------------------------------

Github user kambatla commented on a diff in the pull request:

    https://github.com/apache/hadoop/pull/261#discussion_r134268363
  
    --- Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md
---
    @@ -85,9 +85,9 @@ The allocation file must be in XML format. The format contains five
types of ele
     
         * **minResources**: minimum resources the queue is entitled to, in the form "X mb,
Y vcores". For the single-resource fairness policy, the vcores value is ignored. If a queue's
minimum share is not satisfied, it will be offered available resources before any other queue
under the same parent. Under the single-resource fairness policy, a queue is considered unsatisfied
if its memory usage is below its minimum memory share. Under dominant resource fairness, a
queue is considered unsatisfied if its usage for its dominant resource with respect to the
cluster capacity is below its minimum share for that resource. If multiple queues are unsatisfied
in this situation, resources go to the queue with the smallest ratio between relevant resource
usage and minimum. Note that it is possible that a queue that is below its minimum may not
immediately get up to its minimum when it submits an application, because already-running
jobs may be using those resources.
     
    -    * **maxResources**: maximum resources a queue is allowed, in the form "X mb, Y vcores".
A queue will never be assigned a container that would put its aggregate usage over this limit.
    +    * **maxResources**: maximum resources a queue is allowed, in the form "X mb, Y vcores".
A queue will never be assigned a container that would put its aggregate usage over this limit.
Set it to a percentage of all cluster resources in form of "50%" or "50% memory, 60% vcore".
    --- End diff --
    
    How about:
    
    **maxResources**: maximum resources a queue is allocated, expressed either in absolute
values (X mb, Y vcores) or as a percentage of the cluster resources (X% memory, Y% cpu). A
queue will not be assigned a container that would put its aggregate usage over this limit.



> add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
> -----------------------------------------------------------------------------------------
>
>                 Key: YARN-2162
>                 URL: https://issues.apache.org/jira/browse/YARN-2162
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: fairscheduler, scheduler
>            Reporter: Ashwin Shankar
>            Assignee: Yufei Gu
>              Labels: scheduler
>         Attachments: YARN-2162.001.patch, YARN-2162.002.patch
>
>
> minResources and maxResources in fair scheduler configs are expressed in terms of absolute
numbers X mb, Y vcores. 
> As a result, when we expand or shrink our hadoop cluster, we need to recalculate and
change minResources/maxResources accordingly, which is pretty inconvenient.
> We can circumvent this problem if we can optionally configure these properties in terms
of percentage of cluster capacity. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message