hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sunil G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3215) Respect labels in CapacityScheduler when computing headroom
Date Thu, 21 Jan 2016 14:07:39 GMT

    [ https://issues.apache.org/jira/browse/YARN-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110625#comment-15110625
] 

Sunil G commented on YARN-3215:
-------------------------------

Thanks [~Naganarasimha Garla] for sharing the patch.
{{headroom}} will be mostly less than {{clusterFreePartitionResource}}, and this new min helps
to ensure that we do not over-promise. Approach and patch looks fine for me. 

As discussed offline, One general improvement can be like,

{code}
    Resource queueCapacity =
        Resources.multiplyAndNormalizeUp(resourceCalculator,
            labelManager.getResourceByLabel(nodePartition, clusterResource),
            queueCapacities.getAbsoluteCapacity(nodePartition),
            minimumAllocation);
    Resource queueMaxCapacity =
        Resources.multiplyAndNormalizeUp(resourceCalculator,
            labelManager.getResourceByLabel(nodePartition, clusterResource),
            queueCapacities.getAbsoluteMaximumCapacity(nodePartition),
            minimumAllocation);
{code}

If we can have this values pre-computed, it will be much better. This can  be changed only
when node is added/removed etc. In all other cases, we can just use it via a read-only interface.
Can we have this inside QueueCapacities. Thoughts?


> Respect labels in CapacityScheduler when computing headroom
> -----------------------------------------------------------
>
>                 Key: YARN-3215
>                 URL: https://issues.apache.org/jira/browse/YARN-3215
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacityscheduler
>            Reporter: Wangda Tan
>            Assignee: Naganarasimha G R
>         Attachments: YARN-3215.v1.001.patch, YARN-3215.v2.001.patch, YARN-3215.v2.002.patch
>
>
> In existing CapacityScheduler, when computing headroom of an application, it will only
consider "non-labeled" nodes of this application.
> But it is possible the application is asking for labeled resources, so headroom-by-label
(like 5G resource available under node-label=red) is required to get better resource allocation
and avoid deadlocks such as MAPREDUCE-5928.
> This JIRA could involve both API changes (such as adding a label-to-available-resource
map in AllocateResponse) and also internal changes in CapacityScheduler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message