hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Craig Welch (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2496) [YARN-796] Changes for capacity scheduler to support allocate resource respect labels
Date Sat, 20 Sep 2014 04:02:35 GMT

    [ https://issues.apache.org/jira/browse/YARN-2496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14141736#comment-14141736

Craig Welch commented on YARN-2496:

ach, not finished - anyway, re 2 - a particular job may well only be able to use nodes with
one label in a queue, and so if the headroom includes nodes without that label, we'll end
up with another deadlock case where it spins up reducers too early and then can't complete
its maps.  It is definately a valid usecase to have a queue with two lables (a and b, as in
this example) and an app which is consistently requesting only one of those two labels (from
submission to completion...) - perhaps only "a" nodes have the "special resource" it needs
(special hardware capability, etc).  For this reason, headroom should reflect the labels in
the last resource request from the application, not the queue's labels.

(-re 5, I though * could be in requests, if no, then should not be an issue.)

> [YARN-796] Changes for capacity scheduler to support allocate resource respect labels
> -------------------------------------------------------------------------------------
>                 Key: YARN-2496
>                 URL: https://issues.apache.org/jira/browse/YARN-2496
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: YARN-2496.patch, YARN-2496.patch, YARN-2496.patch, YARN-2496.patch
> This JIRA Includes:
> - Add/parse labels option to {{capacity-scheduler.xml}} similar to other options of queue
like capacity/maximum-capacity, etc.
> - Include a "default-label-expression" option in queue config, if an app doesn't specify
label-expression, "default-label-expression" of queue will be used.
> - Check if labels can be accessed by the queue when submit an app with labels-expression
to queue or update ResourceRequest with label-expression
> - Check labels on NM when trying to allocate ResourceRequest on the NM with label-expression
> - Respect  labels when calculate headroom/user-limit

This message was sent by Atlassian JIRA

View raw message