hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-7136) Additional Performance Improvement for Resource Profile Feature
Date Thu, 07 Sep 2017 22:11:00 GMT

     [ https://issues.apache.org/jira/browse/YARN-7136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Wangda Tan updated YARN-7136:
    Attachment: YARN-7136.YARN-3926.011.patch

Thanks [~templedf] for thorough review. 

1): Done 

bq. In Resource.initResourceMap(), is it safe to remove the special casing for vcores that
are more than max int?
This may still required, because {{Resource#getVirtualCores}} is a public API, we need to
maintain compatibility, if a value set to virtual cores > MAX_INT, it may become negative
when it is returned according to implementation at BaseResource: 
  public int getVirtualCores() {
    return (int) vcoresResInfo.getValue();
An alternative approach is do the check in the getVirtualCores implementation, however from
what we can see, vcores should be far away from 32bits int MAX_VALUE. Considering performance
impact, I would like to keep this logic. Concerns? 

3): Done

4): Done

bq. do we expect to be comparing objects to themselves so often that the first if is a performance
Yes, because ref to Resource is pass around frequently. It would be better to keep it as first
bq. Can we please combine the second and third if statements into one?

6): Done 

7): Done

8): Done 

9): Done 

This code will print all resource types, To maintain backward-compatibility, it prints memory/cpu
by default. I added logic to prevent print value==0 items other than memory and vcores.

11): Done 

12): Done

13): Renamed to LightResource, updated Javadocs accordingly, please let me know if you have
better naming suggestions.

Actually readOnlyResource is not useful at all. If it is a cached array, caller can modify
it and screw other callers as well. Since Java doesn't support immutable native array, and
considering performance issue, I would like to keep it as is. Please let me know if you have
any other suggestions.

Attached (011) patch.

> Additional Performance Improvement for Resource Profile Feature
> ---------------------------------------------------------------
>                 Key: YARN-7136
>                 URL: https://issues.apache.org/jira/browse/YARN-7136
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>            Priority: Critical
>         Attachments: YARN-7136.001.patch, YARN-7136.YARN-3926.001.patch, YARN-7136.YARN-3926.002.patch,
YARN-7136.YARN-3926.003.patch, YARN-7136.YARN-3926.004.patch, YARN-7136.YARN-3926.005.patch,
YARN-7136.YARN-3926.006.patch, YARN-7136.YARN-3926.007.patch, YARN-7136.YARN-3926.008.patch,
YARN-7136.YARN-3926.009.patch, YARN-7136.YARN-3926.010.patch, YARN-7136.YARN-3926.011.patch
> This JIRA is plan to add following misc perf improvements:
> 1) Use final int in Resources/ResourceCalculator to cache #known-resource-types. (Significant
> 2) Catch Java's ArrayOutOfBound Exception instead of checking array.length every time.
(Significant improvement).
> 3) Avoid setUnit validation (which is a HashSet lookup) when initialize default Memory/VCores
ResourceInformation (Significant improvement).
> 4) Avoid unnecessary loop array in Resource#toString/hashCode. (Some improvement).
> 5) Removed readOnlyResources in BaseResource. (Minor improvement).
> 6) Removed enum: MandatoryResources, use final integer instead. (Minor improvement).

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message