hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-7136) Additional Performance Improvement for Resource Profile Feature
Date Fri, 01 Sep 2017 18:27:02 GMT

     [ https://issues.apache.org/jira/browse/YARN-7136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Wangda Tan updated YARN-7136:
    Attachment: YARN-7136.YARN-3926.006.patch

Thanks [~sunilg] for updating patch and comments from [~jlowe]. 

bq. Resource's compareTo is now treating all the resources equally which is nice, but that
means for memory and vcores it will be doing string compares on the names, hash set lookups
on the units, and string comparisons on the units before it ever gets to simply comparing
the numbers which is all we really care about for these resources.
Regarding to performance issue, I think it won't cause much issue since BaseResource.compareTo
uses shortcut. 

bq. The ApplicationMasterService can marshal the app's requests into the normalized resources
before feeding the requests to the scheduler so the scheduler doesn't have to ever worry about
units. Worth a followup JIRA? 
Probably a better solution is directly normalize all resource types to default unit specified
in {{resource-types.xml}} when all Resource get initialized. We can have a flag to only do
this in RM side. It is definitely worth to do, do you think is it better to do this after

bq. Similar performance hit for the Resources equals method since it now needs to check names
and units before checking values on memory and vcores which it did not do before.
BaseResource overwrite equals, there're lots of performance issues once we add 3rd resource
to the system. 

bq. Thinking that if an admin suddenly refreshes the resource types to add one the code could
end up indexing off the end of an array argument that was just passed to it before the refresh
Currently resource types are loaded when RM start, so this couldn't happen now. Adding resource
type to a running RM is very tricky, we can spend more time to solve the problem when it is
really needed.

And addressed all other comments. (006)

> Additional Performance Improvement for Resource Profile Feature
> ---------------------------------------------------------------
>                 Key: YARN-7136
>                 URL: https://issues.apache.org/jira/browse/YARN-7136
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>            Priority: Critical
>         Attachments: YARN-7136.001.patch, YARN-7136.YARN-3926.001.patch, YARN-7136.YARN-3926.002.patch,
YARN-7136.YARN-3926.003.patch, YARN-7136.YARN-3926.004.patch, YARN-7136.YARN-3926.005.patch,
> This JIRA is plan to add following misc perf improvements:
> 1) Use final int in Resources/ResourceCalculator to cache #known-resource-types. (Significant
> 2) Catch Java's ArrayOutOfBound Exception instead of checking array.length every time.
(Significant improvement).
> 3) Avoid setUnit validation (which is a HashSet lookup) when initialize default Memory/VCores
ResourceInformation (Significant improvement).
> 4) Avoid unnecessary loop array in Resource#toString/hashCode. (Some improvement).
> 5) Removed readOnlyResources in BaseResource. (Minor improvement).
> 6) Removed enum: MandatoryResources, use final integer instead. (Minor improvement).

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message