hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Naganarasimha G R (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3409) Add constraint node labels
Date Tue, 06 Dec 2016 00:34:59 GMT

    [ https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15723856#comment-15723856

Naganarasimha G R commented on YARN-3409:

Thanks [~wangda] for the quick feedback,
bq. I still think constraint is one special kind of label, we should reuse existing APIs:

IMHO It depends on how we conclude on whether we require *constraint type*, lets further discuss
on it and then come back to this. and further i cross verified, my bad we support replace
nodelabel mappings and adding cluster node labels through REST.

bq. Do we really need define type of the constraint? 
In the example which i had given software version x.y.z it always doesn't have the numerical
digits to be padded for example {{r2 and r19}}, so ascii comparison might fail, this is one
such example we had faced earlier, but based on scenario it can vary, but if we have a custom
type and define how to compare it would be better to adapt anything in future.

bq. I think we can do late-binding it. 
May be value for the node's constraint is given as {{Long}} and in the expression we give
it as {{Double}} or vice versa, then evaluation will lead to run time failures right ? or
for each evaluation we need to check and define what is the better match or how to compare
two different kinds of values. Late binding would be *too much costly/expensive* when each
request has a constraint expression. As we need to almost validate for such scenarios for
each node against the constraint expression of a resource request in each schedule cycle.
In the approach which i have specified in the patch its almost static comparison of long to
long, or double to double, or type to type, as the type is predetermined type for the Node's
value and the RHS value of the expression.

bq. sometimes we need compare it (like version > and version < and
sometimes we need do string much (like version ~ 2.3.4.*)
bq. Maybe we can support "like" operator as well, it will compare string with given regex.
Agree i too was thinking about supporting regex for the the String Constraint type. further
it would be better to have for constraint Name too, which will be helpful for Boolean type.

bq. We only need support one set of syntax (for example, equal is "==", but "eq" will not
be a valid operation name)
Yeah this is not a basic requirement, it would have been required if we directly use it this
expression in REST api's but we usually use as part of ResourceRequest object, but just thought
in case we want to support dynamic Constraint expression modification for a RR in future.

I would like to get more feedback on the constraint type from others too, hope to get early
cc [~kkaranasos] [~devaraj.k] [~cchen317] [~grey]  and others in the subscriber's list.

> Add constraint node labels
> --------------------------
>                 Key: YARN-3409
>                 URL: https://issues.apache.org/jira/browse/YARN-3409
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: api, capacityscheduler, client
>            Reporter: Wangda Tan
>            Assignee: Naganarasimha G R
>         Attachments: Constraint-Node-Labels-Requirements-Design-doc_v1.pdf, YARN-3409.WIP.001.patch
> Specify only one label for each node (IAW, partition a cluster) is a way to determinate
how resources of a special set of nodes could be shared by a group of entities (like teams,
departments, etc.). Partitions of a cluster has following characteristics:
> - Cluster divided to several disjoint sub clusters.
> - ACL/priority can apply on partition (Only market team / marke team has priority to
use the partition).
> - Percentage of capacities can apply on partition (Market team has 40% minimum capacity
and Dev team has 60% of minimum capacity of the partition).
> Constraints are orthogonal to partition, they’re describing attributes of node’s
hardware/software just for affinity. Some example of constraints:
> - glibc version
> - JDK version
> - Type of CPU (x86_64/i686)
> - Type of OS (windows, linux, etc.)
> With this, application can be able to ask for resource has (glibc.version >= 2.20
&& JDK.version >= 8u20 && x86_64).

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message