hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sunil G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3215) Respect labels in CapacityScheduler when computing headroom
Date Thu, 14 Jan 2016 15:01:39 GMT

    [ https://issues.apache.org/jira/browse/YARN-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15098219#comment-15098219

Sunil G commented on YARN-3215:

Hi [~Naganarasimha Garla]
As discussed offline, I will try explaining the problem more clearly.

{{getQueueMaxResource}} considers {{queueCapacities.getAbsoluteMaximumCapacity}} to calculate
max possible resource limit  per-label. As per the approach, we try to find the difference
with this limit to actual resource used for that label (in a queue) to get the headroom. As
we consider absolute max capacity, there are chances that this limit may reach more than 100%
. (on possibility is that same label is used in another queue).
for eg:
label X is configured in queueA.
             - capacity is 30%
             - max capacity is 60%

label X is configured in queueB.
             - capacity is 70%
             - max capcity is 80%

IN this case, its possible that we may get more than 100% for label X resource limit. So headroom
calculated per queue may be more than what is already available. I think this is not very
major problem, but its better to solved by putting a max cap with total resource limit per
label vs its used limit. Thoughts?

> Respect labels in CapacityScheduler when computing headroom
> -----------------------------------------------------------
>                 Key: YARN-3215
>                 URL: https://issues.apache.org/jira/browse/YARN-3215
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacityscheduler
>            Reporter: Wangda Tan
>            Assignee: Naganarasimha G R
>         Attachments: YARN-3215.v1.001.patch
> In existing CapacityScheduler, when computing headroom of an application, it will only
consider "non-labeled" nodes of this application.
> But it is possible the application is asking for labeled resources, so headroom-by-label
(like 5G resource available under node-label=red) is required to get better resource allocation
and avoid deadlocks such as MAPREDUCE-5928.
> This JIRA could involve both API changes (such as adding a label-to-available-resource
map in AllocateResponse) and also internal changes in CapacityScheduler.

This message was sent by Atlassian JIRA

View raw message