hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3926) Extend the YARN resource model for easier resource-type management and profiles
Date Thu, 16 Jul 2015 06:37:05 GMT

    [ https://issues.apache.org/jira/browse/YARN-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14629288#comment-14629288

Karthik Kambatla commented on YARN-3926:

Thanks a bunch for putting this proposal together, Varun. We are in dire need of improvements
to our resource-model, and the proposal goes a long way in addressing some of these issues.
Huge +1 to this effort. 

Comments on the proposal itself:
# There is a significant overlap between resource-types.xml and node-resources.xml. It would
be nice to consolidate at least these parts. 
# Can we avoid the mismatch between the resource types on RM and NM altogether?
# Can we avoid different restart paths for adding and removing resources? 
# Really like the concise configs proposed at the end of the document. 

What do you think of the following modifications to the proposal to address above wishes?
I have clearly not thought as much before making these suggestions, so please feel free to
shoot them down. 
# How about calling them yarn.resource-types, yarn.resource-types.memory.*, yarn.resource-types.cpu.*.
Further, memory/cpu specific configs could be made simpler per the suggestions later in the
# yarn.scheduler.resource-types is a subset of yarn.resource-types, and captures the resource-types
the scheduler supports. This could be in yarn-site on RM.
# yarn.nodemanager.resource-types.monitored and yarn.nodemanager.resource-types.enforced also
are subsets of yarn.resource-types and could define the resources the NM monitors and enforces
respectively. These could be in yarn-site on the NM. I understand isolation is out of scope
here, but would be nice to have configs that lend themselves to future work.
# yarn.nodemanager.[resources|resource-types].available could be a map where each key should
be an entry in yarn.resource-types. 

You mention capturing node-labels etc. similarly. Could you elaborate on your thoughts, at
least informally? Would be super nice to have a path in mind even if we were to do as follow-up

> Extend the YARN resource model for easier resource-type management and profiles
> -------------------------------------------------------------------------------
>                 Key: YARN-3926
>                 URL: https://issues.apache.org/jira/browse/YARN-3926
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: nodemanager, resourcemanager
>            Reporter: Varun Vasudev
>            Assignee: Varun Vasudev
>         Attachments: Proposal for modifying resource model and profiles.pdf
> Currently, there are efforts to add support for various resource-types such as disk(YARN-2139),
network(YARN-2140), and  HDFS bandwidth(YARN-2681). These efforts all aim to add support for
a new resource type and are fairly involved efforts. In addition, once support is added, it
becomes harder for users to specify the resources they need. All existing jobs have to be
modified, or have to use the minimum allocation.
> This ticket is a proposal to extend the YARN resource model to a more flexible model
which makes it easier to support additional resource-types. It also considers the related
aspect of “resource profiles” which allow users to easily specify the various resources
they need for any given container.

This message was sent by Atlassian JIRA

View raw message