hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6789) new api to get all supported resources from RM
Date Wed, 09 Aug 2017 20:49:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-6789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120636#comment-16120636

Wangda Tan commented on YARN-6789:

After a offline discussion with [~sunilg], I think we discovered more issues for unit in Resource

The "unit" creates several issues:
- Behavior of the branch is: if unit of a given resource information is not set, default unit
which configured in resource-type.cfg will be used to initialize containers ({{Resource.newInstance}});
and unit will be untouched for PB record initialization. ({{ResourcePBImpl(ResourceProto proto)}})
- However, if we have AM runs with old code (which doesn't have YARN-3926 logics), it will
send resource PB record to RM without unit on the wire. So YARN RM thinks the coming memory
value with empty UNIT (which is B). This is an incompatible behavior. 
- Secondly, as I commented above, "unit" inside ResourceTypeInfo is very confusing: a. it
is not minimum unit. b. it is not default unit, since it won't affect "default unit" inside
AM, it is just default unit inside RM, which AM should not be interested. c. it is not a "suggested/preferred
unit", because it doesn't make sense as well.
- In addition, It creates performance issue as well since all Resource operations need convert
to the same unit.

My personal preference is completely removing unit from ResourceInformation. And unit of ResourceType
means unit of given resource type. For example, resource.types.memory.unit = MB. It will be
majorly used for UI displaying. Units of known resource types including vcores/memory will
be hard coded in code and cannot changed by setting configuration file, this is majorly for
backward-compatibility. We can provide unit converter as client library for AM/Client to use,
Resource-related classes should not directly use it.


> new api to get all supported resources from RM
> ----------------------------------------------
>                 Key: YARN-6789
>                 URL: https://issues.apache.org/jira/browse/YARN-6789
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>            Reporter: Sunil G
>            Assignee: Sunil G
>         Attachments: YARN-6789-YARN-3926.001.patch
> It will be better to provide an api to get all supported resource types from RM.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message