hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6788) Improve performance of resource profile branch
Date Mon, 10 Jul 2017 18:51:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16080868#comment-16080868
] 

Wangda Tan commented on YARN-6788:
----------------------------------

Thanks [~sunilg], comments:

Resource:
- {{public abstract Integer getResourceIndex(String resource)}} should be internal API, how
about only add it to {{ResourceUtils.getResourceTypeIndex(String resourceName)}}. We should
try to avoid store any local information of resource_type_to_index.

ResourcePBImpl:
- What does the readOnlyResources means?

bq. In Resource class, there are few impls for SimpleResource. I also had an offline talk
with Varun Vasudev. May be its better to move all resource related impl from ResourcePBImpl
to Resource class itself and can keep only proto related changes in PBImpl class. So SimpleResource
could look into memory and cpu resource info objects.
I suggest to not take this shortcut, we have several built-in resource types are adding to
YARN, such as GPU/FPGA, I don't want to get unexpected performance regression after adding
them.

Resources/DominantResourceCalculator (Maybe there're more places could be changed)
Now they're using either {{setResourceValue(name, value)}}, or {{getResourceInformation(rName)}}.
Both of them will do frequent map-looking operations. Instead of doing this, can we add a
public (marked as {{@private}}) APIs to Resource object, which support get/setResourceInformation/Value
with index. Internally we can use it to do computations.
To me all name-related fields should not be used while doing computations. String-based names
should be only used for human-readability, such as UI/message, etc. 

(Not performance related), there're many places in the code hardcoded to use {{set/getVirtualCores}}
and assume there're only 2 resource types. I suggest to review all of them before merge the
branch. 

> Improve performance of resource profile branch
> ----------------------------------------------
>
>                 Key: YARN-6788
>                 URL: https://issues.apache.org/jira/browse/YARN-6788
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>            Reporter: Sunil G
>            Assignee: Sunil G
>            Priority: Blocker
>         Attachments: YARN-6788-YARN-3926.001.patch
>
>
> Currently we could see a 15% performance delta with this branch. 
> Few performance improvements to improve the same.
> Also this patch will handle [comments|https://issues.apache.org/jira/browse/YARN-6761?focusedCommentId=16075418&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16075418]
from [~leftnoteasy].



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message