hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sunil G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3409) Add constraint node labels
Date Fri, 21 Jul 2017 14:38:01 GMT

    [ https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096324#comment-16096324

Sunil G commented on YARN-3409:

Hi [~templedf]
Lemme take the liberty of pointing out some thoughts here.

Currently we have CommonNodeLabelManager which handles node label and other common tasks such
# Api Handling
# Storage and mappings
# Recovery

For internal scheduler access point, a derivative version of  RMNodeLabelManager is used.
All together, as you mentioned we might need to reinvent whole wheel here. Ideally we would
be looking a Common base class to handle recovery, storage and basic mappings. NodeLabel specific
manager could derive from this base class to handle internal node label specific apis etc.
Similarly for attributes, we can have a RMNodeAttribute manage derived from the common class.
In a nutshell, there are some pieces in node label and attributes which are different and
it could be handled respectively. 

About the point +"value for the label could change between when the allocation is made and
when the AM launches the container"+ , I think it is kind of handled now in node label. We
could see something like containers which are scheduler based on old attribute and still running,
and new set of containers which are scheduled based on new attribute. If container demand
for new attribute is more on that node, all old style containers could be marked for preemption.
In fact, we do same now for node label (when partition of node is changed runtime). Is this
a possible way of handling same. Thoughts?
I think [~naganarasimha_gr@apache.org] also has more thoughts on same. Please correct me if
I am wrong. Thank You.

> Add constraint node labels
> --------------------------
>                 Key: YARN-3409
>                 URL: https://issues.apache.org/jira/browse/YARN-3409
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: api, capacityscheduler, client
>            Reporter: Wangda Tan
>            Assignee: Naganarasimha G R
>         Attachments: Constraint-Node-Labels-Requirements-Design-doc_v1.pdf, YARN-3409.WIP.001.patch
> Specify only one label for each node (IAW, partition a cluster) is a way to determinate
how resources of a special set of nodes could be shared by a group of entities (like teams,
departments, etc.). Partitions of a cluster has following characteristics:
> - Cluster divided to several disjoint sub clusters.
> - ACL/priority can apply on partition (Only market team / marke team has priority to
use the partition).
> - Percentage of capacities can apply on partition (Market team has 40% minimum capacity
and Dev team has 60% of minimum capacity of the partition).
> Constraints are orthogonal to partition, they’re describing attributes of node’s
hardware/software just for affinity. Some example of constraints:
> - glibc version
> - JDK version
> - Type of CPU (x86_64/i686)
> - Type of OS (windows, linux, etc.)
> With this, application can be able to ask for resource has (glibc.version >= 2.20
&& JDK.version >= 8u20 && x86_64).

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message