hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun Suresh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3926) Extend the YARN resource model for easier resource-type management and profiles
Date Sun, 26 Jul 2015 06:36:05 GMT

    [ https://issues.apache.org/jira/browse/YARN-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14641865#comment-14641865

Arun Suresh commented on YARN-3926:

Thanks for the proposal [~vvasudev] !! Interesting stuff..

Couple of comments from my first read of the proposal :

# Instead of Resource.newInstance(Map<ResourceTypeInformation, Long>), can we use the
builder pattern something like so :
ResourceBuilder.dimension(ResourceTypeInformation t1).value(Long v1)
                           .dimension(ResourceTypeInformation t2).value(Long v2)
# The proposal states that if there is a mismatch between what the "resource-types.xml”
contains and what the NM reports, it should shut down. My opinion is that node shut-down should
happen only if Node reports less number of types / does not have all “enabled” types in
resource-types.xml : same rationale as why nodes should not care if the resource type is “enabled”
or not. If node reports more types, that dimension is just ignored. Also, in the section where
you talk about adding/removing types, you mentioned that the NM should be upgraded first..
in which case it will start reporting a new type of resource.. and it should be accepted by
the RM.
# Instead of having to explicitly mark a resource as “countable”, can’t we just assume
thats the default and instead require “uncountable” types to be explicitly specified (once
we start supporting it)
# I really like the Profiles idea… In the profile Section, do we really need a separate
“yarn.scheduler.profile…name” ? can’t we just set “yarn.scheduler.profiles” to
be “minimum,maximum,default,small,large” etc ?

> Extend the YARN resource model for easier resource-type management and profiles
> -------------------------------------------------------------------------------
>                 Key: YARN-3926
>                 URL: https://issues.apache.org/jira/browse/YARN-3926
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: nodemanager, resourcemanager
>            Reporter: Varun Vasudev
>            Assignee: Varun Vasudev
>         Attachments: Proposal for modifying resource model and profiles.pdf
> Currently, there are efforts to add support for various resource-types such as disk(YARN-2139),
network(YARN-2140), and  HDFS bandwidth(YARN-2681). These efforts all aim to add support for
a new resource type and are fairly involved efforts. In addition, once support is added, it
becomes harder for users to specify the resources they need. All existing jobs have to be
modified, or have to use the minimum allocation.
> This ticket is a proposal to extend the YARN resource model to a more flexible model
which makes it easier to support additional resource-types. It also considers the related
aspect of “resource profiles” which allow users to easily specify the various resources
they need for any given container.

This message was sent by Atlassian JIRA

View raw message