hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "patrick white (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3215) Respect labels in CapacityScheduler when computing headroom
Date Tue, 10 Mar 2015 16:23:38 GMT

    [ https://issues.apache.org/jira/browse/YARN-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14355145#comment-14355145

patrick white commented on YARN-3215:

Hi, i am trying to reproduce the fail case as part of Label feature verification for our usecases,
and so far the headroom calculation appears to behave correctly. Would it be possible to provide
a specific fail scenario for this issue?

There were a number of challenges getting the yarn-site and capacity properties correctly
set, we believe we have those in place now. So with both cases of jobs running on labelled
and non-labelled resources, we are seeing the task execution staying on the correct nodes
(a labelled job will only task out to matching-labelled nodes, a non-labelled job will not
task to labelled nodes) and the headroom calc from AM logs show headroom memory dropping to
0 within 5 seconds of job start. This is observed even with small-capacity run queues for
the jobs and having 'slowstart' set to 0.

> Respect labels in CapacityScheduler when computing headroom
> -----------------------------------------------------------
>                 Key: YARN-3215
>                 URL: https://issues.apache.org/jira/browse/YARN-3215
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacityscheduler
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
> In existing CapacityScheduler, when computing headroom of an application, it will only
consider "non-labeled" nodes of this application.
> But it is possible the application is asking for labeled resources, so headroom-by-label
(like 5G resource available under node-label=red) is required to get better resource allocation
and avoid deadlocks such as MAPREDUCE-5928.
> This JIRA could involve both API changes (such as adding a label-to-available-resource
map in AllocateResponse) and also internal changes in CapacityScheduler.

This message was sent by Atlassian JIRA

View raw message