hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Naganarasimha G R (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3409) Support Node Attribute functionality
Date Sat, 22 Jul 2017 00:23:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16097025#comment-16097025

Naganarasimha G R commented on YARN-3409:

Thanks [~daniel@cloudera.com], [~grey], [~sunilg] & [~wangda] for sharing your view points,
We (Huawei folks) too had discussion in the same lines and in YARN we had already created
a concept of "NodeLabels/ NodePartitions" which was not inline with putting everything into
resource. So my view was to have it as separate in terms consistency and clarity. IMHO it
would be more intuitive for users to understand it, if not clubbed as resource.

bq.  "value for the label could change between when the allocation is made and when the AM
launches the container" ,
As [~wangda] corrected its not explicitly triggering Preemption but based on demand it gets
triggered, but in case of attributes may be we need to rethink on it as the attributes might
impact the ability of execution of a container. Assume a version of library is mapped as a
attribute and containers have been allocated based on it , now if the library gets updated
and thus the attribute's value. So would it be better to validate the expression of the containers
(resource request) running on the node or let it take the natural course(fail and get relaunched)
? Earlier in discussion with [~sunilg] we were thinking of preemption based on validation
of expression.

> Support Node Attribute functionality
> ------------------------------------
>                 Key: YARN-3409
>                 URL: https://issues.apache.org/jira/browse/YARN-3409
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: api, capacityscheduler, client
>            Reporter: Wangda Tan
>            Assignee: Naganarasimha G R
>         Attachments: Constraint-Node-Labels-Requirements-Design-doc_v1.pdf, YARN-3409.WIP.001.patch
> Specify only one label for each node (IAW, partition a cluster) is a way to determinate
how resources of a special set of nodes could be shared by a group of entities (like teams,
departments, etc.). Partitions of a cluster has following characteristics:
> - Cluster divided to several disjoint sub clusters.
> - ACL/priority can apply on partition (Only market team / marke team has priority to
use the partition).
> - Percentage of capacities can apply on partition (Market team has 40% minimum capacity
and Dev team has 60% of minimum capacity of the partition).
> Constraints are orthogonal to partition, they’re describing attributes of node’s
hardware/software just for affinity. Some example of constraints:
> - glibc version
> - JDK version
> - Type of CPU (x86_64/i686)
> - Type of OS (windows, linux, etc.)
> With this, application can be able to ask for resource has (glibc.version >= 2.20
&& JDK.version >= 8u20 && x86_64).

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message