hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3214) Add non-exclusive node labels
Date Wed, 25 Mar 2015 01:15:53 GMT

    [ https://issues.apache.org/jira/browse/YARN-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14379092#comment-14379092
] 

Wangda Tan commented on YARN-3214:
----------------------------------

Hi [~lohit],
Thanks for review the doc, for your comments:
bq. If so, it would become too restrictive. Labels on nodes can be seen in multiple dimension
(from app's resource, machine resource and also usecase resouce, eg backfill jobs are placed
on specific set of nodes). In those cases we should have ability to have multiple labels on
node
Yes, now we only support one label for each node (partition). We temporarily support only
one for each node is, if we have multiple labels on each node, it will hard to do resource
planning (like what we did, we can say queue-A can use 40% of label-X and queue-B can use
60% of label-X). Assume a node with label-X and label-Y, and its resource is 10G, it will
be hard to say the node has 10G resource of (X+Y) OR 10G resource of X and Y. This also makes
preemption hard to do. A tradeoff is, if we don't plan resource share (or capacity) on node-labels,
some resource could be wasted and queues can be starved when they still under their configured
capacity.
Multiple labels on node (we call this constraint) is in design stage, we've some thoughts
about it, and will push it to community once it get a better share -- should not take too
long.

bq. Also, in the documents there is mention of scheduling apps without any labels being scheduled
on labeled nodes if resources are idle. Does that also cover apps which could have different
label other than A/B, but still have a label be placed on these nodes when there is free resources
available?
No, it will only try to allocate non-labeled requests to labeled nodes, if a resource request
explicitly asks node label, we will only allocate corresponding labeled resource for it.




> Add non-exclusive node labels 
> ------------------------------
>
>                 Key: YARN-3214
>                 URL: https://issues.apache.org/jira/browse/YARN-3214
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacityscheduler, resourcemanager
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: Non-exclusive-Node-Partition-Design.pdf
>
>
> Currently node labels partition the cluster to some sub-clusters so resources cannot
be shared between partitioned cluster. 
> With the current implementation of node labels we cannot use the cluster optimally and
the throughput of the cluster will suffer.
> We are proposing adding non-exclusive node labels:
> 1. Labeled apps get the preference on Labeled nodes 
> 2. If there is no ask for labeled resources we can assign those nodes to non labeled
apps
> 3. If there is any future ask for those resources , we will preempt the non labeled apps
and give them back to labeled apps.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message