hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3409) Support Node Attribute functionality
Date Tue, 25 Jul 2017 00:08:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099292#comment-16099292

Wangda Tan commented on YARN-3409:


Add my thoughts to your questions, [~naganarasimha_gr@apache.org] please add yours if you
think differently. 

bq. It was not clear to me how the newly added node attributes are going to play with existing
node labels. Is the plan to share some code or will it be completely separate?
There're still many differences between partition and attribute, for example, we don't need
queue-acl for node-attribute, and after revisit existing implementation, we may not need to
support multiple NM launched on the same host with different ports as well.
It may share some basic implementations (like node-label-manager), however for the API level
it might be better to have a separate node-attribute-protocol, since adding them to NodeLabelProto
looks too complex.
The main part of "2. API proto changes" should be read as proposal#1, and "Alternate Proposal
1"/"Alternate Proposal 2" should be read as proposal #2 and #3.

bq. Re: how applications will be specifying node attribute constraints.
I think so, right? +Naga.

bq. In the CLI API the replace and update seem a bit confusing to me ...
Regarding to semantics of node attribute CLI, I think we all agree with {{update}} (adding
new constraints or replacing the value of existing ones). Instead of calling it {{update}}
or {{set}}, how about call it {{add}} (which overwrite value if key presents)?
I suggest to keep {{replace}} as a single operation, since it is a superset of {{remove}}
and {{remove+add}}, which we can provide atomic op as well. Also, I think it might be more
frequently used by end users instead of a plain {{remove}} (what is the scenario we need to
clean up all attribute on a node?).
{{node1:att1=val1}} looks better than {{node1=att1:val1}}.

> Support Node Attribute functionality
> ------------------------------------
>                 Key: YARN-3409
>                 URL: https://issues.apache.org/jira/browse/YARN-3409
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: api, capacityscheduler, client
>            Reporter: Wangda Tan
>            Assignee: Naganarasimha G R
>         Attachments: 3409-apiChanges_v2.pdf (4).pdf, Constraint-Node-Labels-Requirements-Design-doc_v1.pdf,
> Specify only one label for each node (IAW, partition a cluster) is a way to determinate
how resources of a special set of nodes could be shared by a group of entities (like teams,
departments, etc.). Partitions of a cluster has following characteristics:
> - Cluster divided to several disjoint sub clusters.
> - ACL/priority can apply on partition (Only market team / marke team has priority to
use the partition).
> - Percentage of capacities can apply on partition (Market team has 40% minimum capacity
and Dev team has 60% of minimum capacity of the partition).
> Attributes are orthogonal to partition, they’re describing features of node’s hardware/software
just for affinity. Some example of attributes:
> - glibc version
> - JDK version
> - Type of CPU (x86_64/i686)
> - Type of OS (windows, linux, etc.)
> With this, application can be able to ask for resource has (glibc.version >= 2.20
&& JDK.version >= 8u20 && x86_64).

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message