hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7159) Normalize unit of resource objects in RM and avoid to do unit conversion in critical path
Date Tue, 24 Oct 2017 19:47:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217556#comment-16217556

Wangda Tan commented on YARN-7159:

Thanks [~manirajv06@gmail.com] for working on this. 

I agree with what [~sunilg] suggested. Instead of doing this on RM's deserialization part,
we should do this in ResourceInformation#newInstance. The reason is: {{Resources}} class by
definition, can be used by map reduce as well. It will not be enough to only change RM's deserialization
path and remove all unit conversions in Resources.

I also agree that changing {{setUnits}} to check unit might be a perf issue since it is invoked
by {{copy}}. Can we add an internal {{setUnitsWithoutChecking}} method, which will be used
by {{copy}}, and when {{setUnits}} is invoked, instead of setting unit, we will change value.
For example, assume a resource information's value is 100, unit is MB (which is same as {{ResourceUtils.getUnitsOfResourceXXX}}),
if user calls setUnit(GB). Instead of changing unit to GB, we will keep the unit to MB and
change value to 100 * 1024. And we can avoid the expensive {{setUnit}} in all RM's code paths.


> Normalize unit of resource objects in RM and avoid to do unit conversion in critical
> -----------------------------------------------------------------------------------------
>                 Key: YARN-7159
>                 URL: https://issues.apache.org/jira/browse/YARN-7159
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>            Priority: Critical
>         Attachments: YARN-7159.001.patch, YARN-7159.002.patch
> Currently resource conversion could happen in critical code path when different unit
is specified by client. This could impact performance and throughput of RM a lot. We should
do unit normalization when resource passed to RM and avoid expensive unit conversion every

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message