hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jian Fang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
Date Sun, 20 Jul 2014 20:39:42 GMT

    [ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14068041#comment-14068041
] 

Jian Fang commented on YARN-796:
--------------------------------

As Sandy pointed out, seems the scope becomes bigger and bigger. Take our use case as an example,
we initial only need to restrict Application masters not be assigned to some nodes such as
spot instances in EC2. In our design, we only added the following parameters

yarn.label.enabled
yarn.nodemanager.labels
yarn.app.mapreduce.am.labels

to yarn-site.xml and then modified hadoop code. This function works now. With the current
proposal, I wonder how long it may take to finish.

I also doubt about the assumption for admin to configure labels for a cluster. Usually a cluster
comes with hundreds or thousands of nodes, how possible for the admin to manually configure
the labels? This type of work can be easily automated by some script or a java process running
on each node to write the labels such as OS, processor, and other parameters to yarn-site.xml
before the cluster is started. This is especially true for clusters in a cloud because everything
is automated there. The admin UI could only be used in some special cases that require human
intervention.

One use case for dynamic labeling  is that we can put a label to a node when we try to shrink
a cluster so that hadoop will not assign tasks to that node any more to give that node some
grace time to be decommissioned. This is most likely to be implemented by a restful API call
from a process that chooses a node to remove based on cluster metrics of the cluster.


> Allow for (admin) labels on nodes and resource-requests
> -------------------------------------------------------
>
>                 Key: YARN-796
>                 URL: https://issues.apache.org/jira/browse/YARN-796
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Arun C Murthy
>            Assignee: Wangda Tan
>         Attachments: LabelBasedScheduling.pdf, Node-labels-Requirements-Design-doc-V1.pdf,
YARN-796.patch
>
>
> It will be useful for admins to specify labels for nodes. Examples of labels are OS,
processor architecture etc.
> We should expose these labels and allow applications to specify labels on resource-requests.
> Obviously we need to support admin operations on adding/removing node labels.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message