hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3409) Add constraint node labels
Date Mon, 05 Dec 2016 19:28:58 GMT

    [ https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15723113#comment-15723113
] 

Wangda Tan commented on YARN-3409:
----------------------------------

[~varun_saxena], [~Naganarasimha],

Thanks for your feedbacks, however I still think constraint is one special kind of label,
we should reuse existing APIs: they share a lots of common fields and internal implementations.
For example we already have fully support of label modifications in REST API, I don't think
adding a full set of node constraint API is a good idea.

We can still make REST API / CLI backward compatible.

And thanks [~Naganarasimha] for the POC patch, I look a very brief look at it. Some comments:
1) Do we really need define type of the constraint? 
I think we can do late-binding it. For example, software version could be: x.y.z, it is not
double and not long, sometimes we need compare it (like version > 2.3.4.5 and version <
2.3.4.9) and sometimes we need do string much (like version ~ 2.3.4.*)
Another benefit is, with late-binding, we don't need to add node constraint upfront. Node
constraint will be added/removed to the cluster when we change constraint on node, but they
should not conflict with known partition name. 
With this, we can avoid type conflicting as well. In your existing design, if constraint-A
is defined as long, but on node1 admin set a double value, it will report error. Instead,
if we store everything as string, and evaluate it according to constraint expression, it should
be more easier to be used.

2) For operations
- We only need support one set of syntax (for example, equal is "==", but "eq" will not be
a valid operation name)
- Maybe we can support "like" operator as well, it will compare string with given regex. 


> Add constraint node labels
> --------------------------
>
>                 Key: YARN-3409
>                 URL: https://issues.apache.org/jira/browse/YARN-3409
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: api, capacityscheduler, client
>            Reporter: Wangda Tan
>            Assignee: Naganarasimha G R
>         Attachments: Constraint-Node-Labels-Requirements-Design-doc_v1.pdf, YARN-3409.WIP.001.patch
>
>
> Specify only one label for each node (IAW, partition a cluster) is a way to determinate
how resources of a special set of nodes could be shared by a group of entities (like teams,
departments, etc.). Partitions of a cluster has following characteristics:
> - Cluster divided to several disjoint sub clusters.
> - ACL/priority can apply on partition (Only market team / marke team has priority to
use the partition).
> - Percentage of capacities can apply on partition (Market team has 40% minimum capacity
and Dev team has 60% of minimum capacity of the partition).
> Constraints are orthogonal to partition, they’re describing attributes of node’s
hardware/software just for affinity. Some example of constraints:
> - glibc version
> - JDK version
> - Type of CPU (x86_64/i686)
> - Type of OS (windows, linux, etc.)
> With this, application can be able to ask for resource has (glibc.version >= 2.20
&& JDK.version >= 8u20 && x86_64).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message