hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Naganarasimha G R (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3409) Support Node Attribute functionality
Date Tue, 25 Jul 2017 02:21:03 GMT

    [ https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099402#comment-16099402
] 

Naganarasimha G R commented on YARN-3409:
-----------------------------------------

Thanks [~wangda] & [~kkaranasos] for sharing your comments,
bq. NodeIdToAttributesProto actually be NodeToAttributesProto
agree, wanted to further discuss on this, Inside the proto i was sending across NodeId, I
think we can change it to string, Also even if user specify the hostname/IP:port format do
we need to consider picking "hostname/IP" only, thoughts ?

bq. It was not clear to me how the newly added node attributes are going to play with existing
node labels. Is the plan to share some code or will it be completely separate?
Agree with Wangda's reply on this, there are still lot of differences even in the way we are
going use CommonNodeLabelManager for a Attribute and a Partition. Will try to bring the hierarchy
in it so that some common things can be reused.

bq. The main part of "2. API proto changes" should be read as proposal#1, and "Alternate Proposal
1"/"Alternate Proposal 2" should be read as proposal #2 and #3.
Yep, was trying to capture it as alternates to the proposed proposal. 

bq. is the plan to use the new constraints API we are introducing in YARN-6593?
Yes, Its same as per the earlier discussion, but did not get a chance to completely review
it. Will have a look at it and if any issues will point out there.

bq. In the CLI API the replace and update seem a bit confusing to me ...
Agree with [~wangda]'s point, and my preference would be {{add}}

bq. Sounds a little ambiguous as it does not directly look like the existing attributes on
the node will be removed, but we can make this clear in the description of the command.
Agree, i think i have captured in the CLI patch and if required will update more

bq. node1:att1=val1 looks better than node1=att1:val1.
IMHO i would prefer the latter one itself for following reasons 
1. It might be general scenario to specify multiple mapping of attribute value pairs to a
given node and it would be less input for users. 
2. With earlier notation, port was specified after the {{":"}} before {{"="}} for labels,
so it would less intuitive for users to use the new format  
3. With attribute type it might not sound very effective. like {{node1:att1=val1:type}} or
{{node1:att1:type=val1}} 




> Support Node Attribute functionality
> ------------------------------------
>
>                 Key: YARN-3409
>                 URL: https://issues.apache.org/jira/browse/YARN-3409
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: api, capacityscheduler, client
>            Reporter: Wangda Tan
>            Assignee: Naganarasimha G R
>         Attachments: 3409-apiChanges_v2.pdf (4).pdf, Constraint-Node-Labels-Requirements-Design-doc_v1.pdf,
YARN-3409.WIP.001.patch
>
>
> Specify only one label for each node (IAW, partition a cluster) is a way to determinate
how resources of a special set of nodes could be shared by a group of entities (like teams,
departments, etc.). Partitions of a cluster has following characteristics:
> - Cluster divided to several disjoint sub clusters.
> - ACL/priority can apply on partition (Only market team / marke team has priority to
use the partition).
> - Percentage of capacities can apply on partition (Market team has 40% minimum capacity
and Dev team has 60% of minimum capacity of the partition).
> Attributes are orthogonal to partition, they’re describing features of node’s hardware/software
just for affinity. Some example of attributes:
> - glibc version
> - JDK version
> - Type of CPU (x86_64/i686)
> - Type of OS (windows, linux, etc.)
> With this, application can be able to ask for resource has (glibc.version >= 2.20
&& JDK.version >= 8u20 && x86_64).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message