hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6788) Improve performance of resource profile branch
Date Thu, 13 Jul 2017 17:47:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16086086#comment-16086086
] 

Wangda Tan commented on YARN-6788:
----------------------------------

Thanks [~sunilg] for updating the patch, some comments: 

AbstractResource: 
- It's better to rename it to BaseResource since it is not abstract.
- {{long memory, long vcores}} are not read in constructor.
- I think longer term goal is to extend AbstractResource to more than 2 resource types correct?
If you agree so, could you add a TODO comment in the code?  

And existing patch still has lots of map looking operations, I'm not sure how this may affect
performance. 

Copy-paste my comment to here: 
{code}
Resources/DominantResourceCalculator (Maybe there're more places could be changed)
Now they're using either setResourceValue(name, value), or getResourceInformation(rName).
Both of them will do frequent map-looking operations. Instead of doing this, can we add a
public (marked as @private) APIs to Resource object, which support get/setResourceInformation/Value
with index. Internally we can use it to do computations.
To me all name-related fields should not be used while doing computations. String-based names
should be only used for human-readability, such as UI/message, etc.
{code}

Considering size of the patch, I think we can get some performance results before changing
the code.

> Improve performance of resource profile branch
> ----------------------------------------------
>
>                 Key: YARN-6788
>                 URL: https://issues.apache.org/jira/browse/YARN-6788
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>            Reporter: Sunil G
>            Assignee: Sunil G
>            Priority: Blocker
>         Attachments: YARN-6788-YARN-3926.001.patch, YARN-6788-YARN-3926.002.patch, YARN-6788-YARN-3926.003.patch,
YARN-6788-YARN-3926.004.patch
>
>
> Currently we could see a 15% performance delta with this branch. 
> Few performance improvements to improve the same.
> Also this patch will handle [comments|https://issues.apache.org/jira/browse/YARN-6761?focusedCommentId=16075418&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16075418]
from [~leftnoteasy].



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message