hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun Suresh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3926) Extend the YARN resource model for easier resource-type management and profiles
Date Tue, 15 Mar 2016 20:01:33 GMT

    [ https://issues.apache.org/jira/browse/YARN-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15196092#comment-15196092

Arun Suresh commented on YARN-3926:

[~vvasudev], I was wondering if the possibility of allowing an NM to broadcast newly acquired
resources is something we can factor in.

* Assume, as per the design doc, each NM starts up with an initial *node-resource.xml* which
talks about the available resource it has initially, which consists of a sub-set of resource
types known to the RM. Any resource type unknown to the RM is simply ignored by the RM when
making scheduling decisions.
* At some point, we allow either the admin or via some self-discovery mechanism on the NM
to add new resource types and advertise to the RM a resource update (ofcourse, these types
should be know a-priori by the RM via the *resource-types.xml*... or we should probably add
Admin API on the RM to update/add/remove the resource types on the fly)

Thoughts ?

> Extend the YARN resource model for easier resource-type management and profiles
> -------------------------------------------------------------------------------
>                 Key: YARN-3926
>                 URL: https://issues.apache.org/jira/browse/YARN-3926
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: nodemanager, resourcemanager
>            Reporter: Varun Vasudev
>            Assignee: Varun Vasudev
>         Attachments: Proposal for modifying resource model and profiles.pdf
> Currently, there are efforts to add support for various resource-types such as disk(YARN-2139),
network(YARN-2140), and  HDFS bandwidth(YARN-2681). These efforts all aim to add support for
a new resource type and are fairly involved efforts. In addition, once support is added, it
becomes harder for users to specify the resources they need. All existing jobs have to be
modified, or have to use the minimum allocation.
> This ticket is a proposal to extend the YARN resource model to a more flexible model
which makes it easier to support additional resource-types. It also considers the related
aspect of “resource profiles” which allow users to easily specify the various resources
they need for any given container.

This message was sent by Atlassian JIRA

View raw message