hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3153) Capacity Scheduler max AM resource limit for queues is defined as percentage but used as ratio
Date Thu, 12 Feb 2015 00:13:11 GMT

    [ https://issues.apache.org/jira/browse/YARN-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14317255#comment-14317255
] 

Wangda Tan commented on YARN-3153:
----------------------------------

Since most of the capacity-related config are ranged \[0, 100\], maximum-am-resource-percent
should be a part of capacity settings like queue capacity, queue maximum-capacity.

So I propose to make config to be
Global configuration:
{{yarn.scheduler.capacity.maximum-am-capacity-per-queue}}, default is 10 (10%)

Queue configuration:
{{yarn.scheduler.capacity.<queue-path>.maximum-am-capacity}}

And to avoid confusion, we should deprecate:
{{yarn.scheduler.capacity.maximum-am-resource-percent}}
{{yarn.scheduler.capacity.<queue-path>.maximum-am-resource-percent}}

In addition, maximum-am-capacity for queue is inheritable, when admin set a value for max-am
in parent queue, leaf queue will inherit max-am if itself doesn't set.

Sounds like a plan?

> Capacity Scheduler max AM resource limit for queues is defined as percentage but used
as ratio
> ----------------------------------------------------------------------------------------------
>
>                 Key: YARN-3153
>                 URL: https://issues.apache.org/jira/browse/YARN-3153
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>            Priority: Critical
>
> In existing Capacity Scheduler, it can limit max applications running within a queue.
The config is yarn.scheduler.capacity.maximum-am-resource-percent, but actually, it is used
as "ratio", in implementation, it assumes input will be \[0,1\]. So now user can specify it
up to 100, which makes AM can use 100x of queue capacity. We should fix that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message