hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Naganarasimha G R (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3214) Add non-exclusive node labels
Date Thu, 05 Mar 2015 15:03:39 GMT

    [ https://issues.apache.org/jira/browse/YARN-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348839#comment-14348839

Naganarasimha G R commented on YARN-3214:

Hi [~wangda], 
I had a query (not sure whether this is the jira i need to discuss this though)
IIUC when Labels Requirement started we were trying to cater to 2 kinds of requirements
# Similar to the current jira, label the nodes and try to partition the cluster and ensure
few queues/users get particular partition of nodes with high priority.(Multi Tenant scenario)
# Tagging the node with particular labels (like high MEM Nodes, More CPU cores, has more or
particular kind of GPU's,  has particular library version , java version etc...) and trying
to launch apps based on these tags.

Currently it seems like we are only focusing on first kind, and almost not supporting second
one at all (we are not even accepting more than a label for a node), So i was thinking like
we can support 2 kinds of labels; First kind of Labels which we will be able to support capacity
 and second kind of label for tagging. And any given node can have at most one label of the
first kind (one on which capacity can be specified ) and multiple tag kind of labels. App
can specify label expression on tag kind of labels.
Correct me if my understanding is wrong or if there can be still limitations with the above
said approach too.

> Add non-exclusive node labels 
> ------------------------------
>                 Key: YARN-3214
>                 URL: https://issues.apache.org/jira/browse/YARN-3214
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacityscheduler, resourcemanager
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: Non-exclusive-Node-Partition-Design.pdf
> Currently node labels partition the cluster to some sub-clusters so resources cannot
be shared between partitioned cluster. 
> With the current implementation of node labels we cannot use the cluster optimally and
the throughput of the cluster will suffer.
> We are proposing adding non-exclusive node labels:
> 1. Labeled apps get the preference on Labeled nodes 
> 2. If there is no ask for labeled resources we can assign those nodes to non labeled
> 3. If there is any future ask for those resources , we will preempt the non labeled apps
and give them back to labeled apps.

This message was sent by Atlassian JIRA

View raw message