hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "LiangYe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7854) Attach prefixes to different type of node attributes
Date Fri, 02 Feb 2018 04:02:01 GMT

    [ https://issues.apache.org/jira/browse/YARN-7854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349752#comment-16349752
] 

LiangYe commented on YARN-7854:
-------------------------------

In k8s, [Namespace|https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture/namespaces.md]  is
a mechanism to partition resources created by users into logically named group.

 

The  [Namespace|https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/] provides
a unique scope for:
 # named resources (pods, services, replication controllers, etc. to avoid basic naming collisions,
{color:#FF0000}*now not include nodes and persistantVolume*{color})
 # delegated management authority to trusted users
 # ability to limit community resource consumption

In k8s, _Labels_ are key/value pairs that are attached to objects, such as pods, nodes.  Valid
label keys have two segments: an optional prefix and name, separated by a slash ({{/}}).  If
the prefix is omitted, the label Key is presumed to be private to the user. Automated system
components (e.g. {{kube-scheduler}}, {{kube-controller-manager}}, {{kube-apiserver}}, {{kubectl}},
or other third-party automation) which add labels to end-user objects must specify a prefix.
The {{kubernetes.io/}} prefix is reserved for Kubernetes core components.

In k8s, according to different object's labels, there is different affinity, e.g. [node labels——node
affinity|https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduling/nodeaffinity.md],
[pod labels——pod affinity|https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduling/podaffinity.md]. 
Especially [node taints——pod toleration|https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduling/taint-toleration-dedicated.md].

In yarn, according to  YARN-3409   YARN-6592  [#YARN-7800]  and so, node attribute is
like k8s node labels, allocation tags to container is like k8s pod labels but in k8s pod
labels not delivery to nodes for namespace isolation, node constraint is some kind of k8s
taints and tolerations.

So to this Jira issue, to make up prefix for node attributes from different  sources is a
kind of namespace isolation. 

For now, 
 # Centralized: attributes set by users (admin or normal users) can use a prefix like {color:#FF0000}yarn_rm_admin/{color}
 # Distributed: attributes collected by a certain attribute provider on each NM can use a
prefix like {color:#FF0000}yarn_nm/{color}
 # System: some built-in attributes in yarn, set by yarn internal components, e.g scheduler
like {color:#FF0000}yarn_system/{color}

then,

In RM merge from these sources, no need to handle the conflict cases.

In placement constraints for node attributes should specify prefix to indicate the sources/namespaces,
or use yarn_nm/ prefix by default.

[~cheersyang] [~leftnoteasy] [~Naganarasimha]  any comment?  Thanks.

> Attach prefixes to different type of node attributes
> ----------------------------------------------------
>
>                 Key: YARN-7854
>                 URL: https://issues.apache.org/jira/browse/YARN-7854
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: RM
>            Reporter: Weiwei Yang
>            Assignee: LiangYe
>            Priority: Major
>
> There are multiple types of node attributes depending on which source it comes from,
includes
>  # Centralized: attributes set by users (admin or normal users)
>  # Distributed: attributes collected by a certain attribute provider on each NM
>  # System: some built-in attributes in yarn, set by yarn internal components, e.g scheduler
> To better manage these attributes, we introduce the prefix (namespace) concept to the
an attribute. This Jira is opened to figure out how to attach prefixes (automatically/implicitly
or explicitly) to different type of attributes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message