hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3215) Respect labels in CapacityScheduler when computing headroom
Date Wed, 13 Jan 2016 05:09:40 GMT

    [ https://issues.apache.org/jira/browse/YARN-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15095620#comment-15095620

Wangda Tan commented on YARN-3215:


Thanks for reply, I think the optimization (i.e consider the labels only for those resource
requests with container number >0 and hostName = *) may be not required. For example, a
app could ask for resources in waves, at first want 100 containers, and after a while, ask
100 more containers. If scheduler returns headroom only considering partitions have "active"
requests, application may think there's no room for the 100 more containers.

Another reason is, existing applications will ask less partitions. For example MR needs at
most two different partitions for tasks (one for mappers, and one for reducers), which could
less than number of different priorities.

I would suggest to consider unique partition only in this JIRA, this could be optimized in
the future when we have real life requirements.

> Respect labels in CapacityScheduler when computing headroom
> -----------------------------------------------------------
>                 Key: YARN-3215
>                 URL: https://issues.apache.org/jira/browse/YARN-3215
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacityscheduler
>            Reporter: Wangda Tan
>            Assignee: Naganarasimha G R
>         Attachments: YARN-3215.v1.001.patch
> In existing CapacityScheduler, when computing headroom of an application, it will only
consider "non-labeled" nodes of this application.
> But it is possible the application is asking for labeled resources, so headroom-by-label
(like 5G resource available under node-label=red) is required to get better resource allocation
and avoid deadlocks such as MAPREDUCE-5928.
> This JIRA could involve both API changes (such as adding a label-to-available-resource
map in AllocateResponse) and also internal changes in CapacityScheduler.

This message was sent by Atlassian JIRA

View raw message