hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jian Fang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
Date Fri, 23 May 2014 21:01:04 GMT

    [ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14007698#comment-14007698

Jian Fang commented on YARN-796:

I like to add a use case to this JIRA. 

In a cloud environment, hadoop could run in heterogeneous groups of instances.Take Amazon
EMR as an example, usually an EMR hadoop cluster runs in master, core, and task groups, where
the task group could be spot instances that can go away at any time. As a result, we like
to have a tag capability on each node. That is to say, when a node manager starts up, it will
load the tags from the configuration file. Then, the resource manager could refine the scheduling
results based on the tags. 

One good example is that we don't want an application master to be assigned to any spot instance
in a task group because that instance could be taken away by EC2 at any time. 

If hadoop resource could support a tag capability, then we could extend the current scheduling
algorithm to add constraints to not assign the application master to a task node.

We don't really need any admin capability for the tags (but still good to have) since the
tags are static and can be specified in a configuration file, for example yarn-site.xml.

> Allow for (admin) labels on nodes and resource-requests
> -------------------------------------------------------
>                 Key: YARN-796
>                 URL: https://issues.apache.org/jira/browse/YARN-796
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Arun C Murthy
>            Assignee: Wangda Tan
>         Attachments: YARN-796.patch
> It will be useful for admins to specify labels for nodes. Examples of labels are OS,
processor architecture etc.
> We should expose these labels and allow applications to specify labels on resource-requests.
> Obviously we need to support admin operations on adding/removing node labels.

This message was sent by Atlassian JIRA

View raw message