hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1297) Miscellaneous Fair Scheduler speedups
Date Thu, 02 Jan 2014 21:31:50 GMT

    [ https://issues.apache.org/jira/browse/YARN-1297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13860792#comment-13860792

Karthik Kambatla commented on YARN-1297:

First round of comments:
# Would be nice to see what the gains are corresponding to replacing ResourcePBImpl with SimpleResource.
If it is not noticeable, it might be better to leave it as is.
# At a couple of places, instead of modifying the resource usage of a queue this way, it would
be better to add a method to FSQueue that does this. 
+      Resources.addTo(cur.getResourceUsage(), container.getResource());
# I am surprised direct comparisons instead of DefaultResourceCalculator have a noticeable
performance difference. Can we measure the gains due to this change, and drop it if none.
-      Resource minShare1 = Resources.min(RESOURCE_CALCULATOR, null,
-          s1.getMinShare(), s1.getDemand());
+      int minShare1 = Math.min(s1.getMinShare().getMemory(),
+          s1.getDemand().getMemory());
# I am not an expert, but I hear Math#signum is supposed to be optimized for performance.
Just curious - how much did changing this help? 

> Miscellaneous Fair Scheduler speedups
> -------------------------------------
>                 Key: YARN-1297
>                 URL: https://issues.apache.org/jira/browse/YARN-1297
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: scheduler
>            Reporter: Sandy Ryza
>            Assignee: Sandy Ryza
>         Attachments: YARN-1297-1.patch, YARN-1297.patch, YARN-1297.patch
> I ran the Fair Scheduler's core scheduling loop through a profiler to and identified
a bunch of minimally invasive changes that can shave off a few milliseconds.
> The main one is demoting a couple INFO log messages to DEBUG, which brought my benchmark
down from 16000 ms to 6000.
> A few others (which had way less of an impact) were
> * Most of the time in comparisons was being spent in Math.signum.  I switched this to
direct ifs and elses and it halved the percent of time spent in comparisons.
> * I removed some unnecessary instantiations of Resource objects
> * I made it so that queues' usage wasn't calculated from the applications up each time
getResourceUsage was called.

This message was sent by Atlassian JIRA

View raw message