hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
Date Mon, 21 Jul 2014 03:41:40 GMT

    [ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14068163#comment-14068163

Wangda Tan commented on YARN-796:

Hi [~sunilg],
bq. 2. Regarding reservations, how about introducing node-label reservations. Ideas is like,
if an application is lacking resource on a node, it can reserve on that node as well as to
node-label. So when a suitable node update comes from another node in same node-label, can
try allocating container in new node by unreserving from old node.
I think this makes sense, we'd better support this. I will check our current resource reservation/unreservation
logic how to support it, will keep you posted.

bq. 3. My approach was more like have a centralized configuration, but later after some time,
if want to add a new node to cluster, then it can start with a hardcoded label in its yarn-site.
In your approach, we need to use REStful API or admin command to bring this node under one
label. May be while start up itself this node can be set under a label. your thoughts?
I think a problem of mixed centralized/distributed configuration I can see is, it will be
hard to manage them after RM/NM restart -- should we use labels specified in NM config or
our centralized config? I also replied Jian Fang previously about this: https://issues.apache.org/jira/browse/YARN-796?focusedCommentId=14063316&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14063316.
Maybe a workaround is we can define the centralized config all always overwrite distributed
config. E.g. user defined "GPU" in NM config, and admin use RESTful added "FPGA", RM will
serialize both "GPU", "FPGA" into a centralized storage system. And after RM restart or NM
restart, RM will ignore NM config if anything defined in RM. But I still think it's better
to avoid use both of them together.

> Allow for (admin) labels on nodes and resource-requests
> -------------------------------------------------------
>                 Key: YARN-796
>                 URL: https://issues.apache.org/jira/browse/YARN-796
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Arun C Murthy
>            Assignee: Wangda Tan
>         Attachments: LabelBasedScheduling.pdf, Node-labels-Requirements-Design-doc-V1.pdf,
> It will be useful for admins to specify labels for nodes. Examples of labels are OS,
processor architecture etc.
> We should expose these labels and allow applications to specify labels on resource-requests.
> Obviously we need to support admin operations on adding/removing node labels.

This message was sent by Atlassian JIRA

View raw message