Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 1 Sep 2017 18:27:02 +0000 (UTC)
From: "Wangda Tan (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.13098680.1504130903000.11620.1504290422363@Atlassian.JIRA>
In-Reply-To: <JIRA.13098680.1504130903000@Atlassian.JIRA>
References: <JIRA.13098680.1504130903000@Atlassian.JIRA> <JIRA.13098680.1504130903848@jira-lw-us.apache.org>
Subject: [jira] [Updated] (YARN-7136) Additional Performance Improvement for
 Resource Profile Feature
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 01 Sep 2017 18:27:10 -0000


     [ https://issues.apache.org/jira/browse/YARN-7136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wangda Tan updated YARN-7136:
-----------------------------
    Attachment: YARN-7136.YARN-3926.006.patch

Thanks [~sunilg] for updating patch and comments from [~jlowe]. 

bq. Resource's compareTo is now treating all the resources equally which is nice, but that means for memory and vcores it will be doing string compares on the names, hash set lookups on the units, and string comparisons on the units before it ever gets to simply comparing the numbers which is all we really care about for these resources.
Regarding to performance issue, I think it won't cause much issue since BaseResource.compareTo uses shortcut. 

bq. The ApplicationMasterService can marshal the app's requests into the normalized resources before feeding the requests to the scheduler so the scheduler doesn't have to ever worry about units. Worth a followup JIRA? 
Probably a better solution is directly normalize all resource types to default unit specified in {{resource-types.xml}} when all Resource get initialized. We can have a flag to only do this in RM side. It is definitely worth to do, do you think is it better to do this after merge? 

bq. Similar performance hit for the Resources equals method since it now needs to check names and units before checking values on memory and vcores which it did not do before.
BaseResource overwrite equals, there're lots of performance issues once we add 3rd resource to the system. 

bq. Thinking that if an admin suddenly refreshes the resource types to add one the code could end up indexing off the end of an array argument that was just passed to it before the refresh occurred. 
Currently resource types are loaded when RM start, so this couldn't happen now. Adding resource type to a running RM is very tricky, we can spend more time to solve the problem when it is really needed.

And addressed all other comments. (006)

> Additional Performance Improvement for Resource Profile Feature
> ---------------------------------------------------------------
>
>                 Key: YARN-7136
>                 URL: https://issues.apache.org/jira/browse/YARN-7136
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>            Priority: Critical
>         Attachments: YARN-7136.001.patch, YARN-7136.YARN-3926.001.patch, YARN-7136.YARN-3926.002.patch, YARN-7136.YARN-3926.003.patch, YARN-7136.YARN-3926.004.patch, YARN-7136.YARN-3926.005.patch, YARN-7136.YARN-3926.006.patch
>
>
> This JIRA is plan to add following misc perf improvements:
> 1) Use final int in Resources/ResourceCalculator to cache #known-resource-types. (Significant improvement).
> 2) Catch Java's ArrayOutOfBound Exception instead of checking array.length every time. (Significant improvement).
> 3) Avoid setUnit validation (which is a HashSet lookup) when initialize default Memory/VCores ResourceInformation (Significant improvement).
> 4) Avoid unnecessary loop array in Resource#toString/hashCode. (Some improvement).
> 5) Removed readOnlyResources in BaseResource. (Minor improvement).
> 6) Removed enum: MandatoryResources, use final integer instead. (Minor improvement).


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org