hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantinos Karanasos (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-3409) Add constraint node labels
Date Thu, 08 Dec 2016 01:30:59 GMT

    [ https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15730729#comment-15730729
] 

Konstantinos Karanasos edited comment on YARN-3409 at 12/8/16 1:30 AM:
-----------------------------------------------------------------------

Hey guys, apologies for the late reply.
Here are my thoughts...

bq. Add a new field for constraint expression, and also for affnity/anti-affinity (Per suggested
by Kostas). This should have minimum impact to existing features. But after this, the "nodeLabelExpression
becomes a little ambiguous, we may need to deprecate existing nodeLabelExpression.
Agreed with that, with one clarification: do you mean having an extra affinity/anti-affinity
constraint expression or use the same constraint expression? Probably we will need a separate
one.

bq. Extend existing NodeLabel object to support node constraint, we only need two additional
field to support node constraint. 1) isNodeConstraint 2) Value (For example, we can have a
constraint named jdk-verion, and value could be 6/7/8).
I followed your discussion on this and on evaluating the constraints. I also had an offline
discussion with [~chris.douglas].
I will suggest to have an even simpler approach than the one Wangda proposed.
I believe we should have a first version with just boolean expressions, that is, simply request
whether a label exists or not (possibly including negation of boolean expressions). 
In other words, I suggest to have neither a constraint type nor a value.
Let's have a first simple version of (boolean) labels that works. In a future iteration of
this, we can add attributes (i.e., with values) instead of labels.

Having simple labels allows us to bypass the problem of constraint types. Like Wangda says,
constraint types are not really solving the problem of comparing values, given that people
will right their values in different formats. You can also give a look at YARN-44676 for an
efficient boolean expression matcher.
For example, using simple labels, one node can be annotated with label "Java6". Then a task
that requires at least Java 5 can request for a node with "Java5 || Java6". I think that with
our current use cases, this will be sufficient.

Let me know what you think.


was (Author: kkaranasos):
Hey guys, apologies for the late reply.
Here are my thoughts...

bq. Add a new field for constraint expression, and also for affnity/anti-affinity (Per suggested
by Kostas). This should have minimum impact to existing features. But after this, the "nodeLabelExpression
becomes a little ambiguous, we may need to deprecate existing nodeLabelExpression.
Agreed with that, with one clarification: do you mean having an extra affinity/anti-affinity
constraint expression or use the same constraint expression? Probably we will need a separate
one.

bq. Extend existing NodeLabel object to support node constraint, we only need two additional
field to support node constraint. 1) isNodeConstraint 2) Value (For example, we can have a
constraint named jdk-verion, and value could be 6/7/8).
I followed your discussion on this and on evaluating the constraints. I also had an offline
discussion with [~chris.douglas].
I will suggest to have an even simpler approach than the one Wangda proposed.
I believe we should have a first version with just boolean expressions, that is, simply request
whether a label exists or not (possibly including negation of boolean expressions). 
In other words, I suggest to have neither a constraint type nor a value.
Let's have a first simple version of (boolean) labels that works. In a future iteration of
this, we can add attributes (i.e., with values) instead of labels.

Having simple labels allows us to bypass the problem of constraint types. Like Wangda says,
constraint types are not really solving the problem of comparing values, given that people
will right their values in different formats. You can also give a look at YARN-4467 for an
efficient boolean expression matcher.
For example, using simple labels, one node can be annotated with label "Java6". Then a task
that requires at least Java 5 can request for a node with "Java5 || Java6". I think that with
our current use cases, this will be sufficient.

Let me know what you think.

> Add constraint node labels
> --------------------------
>
>                 Key: YARN-3409
>                 URL: https://issues.apache.org/jira/browse/YARN-3409
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: api, capacityscheduler, client
>            Reporter: Wangda Tan
>            Assignee: Naganarasimha G R
>         Attachments: Constraint-Node-Labels-Requirements-Design-doc_v1.pdf, YARN-3409.WIP.001.patch
>
>
> Specify only one label for each node (IAW, partition a cluster) is a way to determinate
how resources of a special set of nodes could be shared by a group of entities (like teams,
departments, etc.). Partitions of a cluster has following characteristics:
> - Cluster divided to several disjoint sub clusters.
> - ACL/priority can apply on partition (Only market team / marke team has priority to
use the partition).
> - Percentage of capacities can apply on partition (Market team has 40% minimum capacity
and Dev team has 60% of minimum capacity of the partition).
> Constraints are orthogonal to partition, they’re describing attributes of node’s
hardware/software just for affinity. Some example of constraints:
> - glibc version
> - JDK version
> - Type of CPU (x86_64/i686)
> - Type of OS (windows, linux, etc.)
> With this, application can be able to ask for resource has (glibc.version >= 2.20
&& JDK.version >= 8u20 && x86_64).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message