hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lei Guo (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3409) Add constraint node labels
Date Wed, 24 May 2017 15:08:04 GMT

    [ https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023067#comment-16023067

Lei Guo commented on YARN-3409:

[~templedf], we also came from HPC background,  and had discussions on similiar approach last
year with [~leftnoteasy]/[~kkaranasos]/others. For scheduling purpose, there are always two
phases: filtering and placement. The filtering phase is to reduce the search scope of candidates;
while the placement phase is to find the best candidate and allocate resources. Both consumable
resource and non-consumable resource could be used in the filtering phase, and only the consumable
resource involved in the second phase. 

For HPC world, we model everything as resource, so no logic duplication in filtering phase,
while the placement phase need some checking on whether the resource requested is consumable.

For Yarn, so far the resource defined in YARN-3926 are consumable resource, and partition
label (close to host partition in HPC) is already introduced for filtering phase, there is
no obvious advantage of modeling constraints as resource like in HPC world. So we concluded
constraints could be modeled similar to label instead of resource for better consistency,
Basically, all Labels (partition label or constraint label) are only for the filtering; and
all resource are consumable.

> Add constraint node labels
> --------------------------
>                 Key: YARN-3409
>                 URL: https://issues.apache.org/jira/browse/YARN-3409
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: api, capacityscheduler, client
>            Reporter: Wangda Tan
>            Assignee: Naganarasimha G R
>         Attachments: Constraint-Node-Labels-Requirements-Design-doc_v1.pdf, YARN-3409.WIP.001.patch
> Specify only one label for each node (IAW, partition a cluster) is a way to determinate
how resources of a special set of nodes could be shared by a group of entities (like teams,
departments, etc.). Partitions of a cluster has following characteristics:
> - Cluster divided to several disjoint sub clusters.
> - ACL/priority can apply on partition (Only market team / marke team has priority to
use the partition).
> - Percentage of capacities can apply on partition (Market team has 40% minimum capacity
and Dev team has 60% of minimum capacity of the partition).
> Constraints are orthogonal to partition, they’re describing attributes of node’s
hardware/software just for affinity. Some example of constraints:
> - glibc version
> - JDK version
> - Type of CPU (x86_64/i686)
> - Type of OS (windows, linux, etc.)
> With this, application can be able to ask for resource has (glibc.version >= 2.20
&& JDK.version >= 8u20 && x86_64).

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message