hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1297) Miscellaneous Fair Scheduler speedups
Date Thu, 21 Apr 2016 23:21:13 GMT

    [ https://issues.apache.org/jira/browse/YARN-1297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15252984#comment-15252984
] 

Karthik Kambatla commented on YARN-1297:
----------------------------------------

Looks like the test failures are because of preemption and runnability logic and test's reliance
on the timing of the queue-usage update. I remember Sandy measured the logging changes themselves
led to good improvement. 

What do you think of doing only the logging changes here and drive the resources change as
part of YARN-4090? 

> Miscellaneous Fair Scheduler speedups
> -------------------------------------
>
>                 Key: YARN-1297
>                 URL: https://issues.apache.org/jira/browse/YARN-1297
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: fairscheduler
>            Reporter: Sandy Ryza
>            Assignee: Yufei Gu
>         Attachments: YARN-1297-1.patch, YARN-1297-2.patch, YARN-1297.005.patch, YARN-1297.3.patch,
YARN-1297.4.patch, YARN-1297.4.patch, YARN-1297.patch, YARN-1297.patch
>
>
> I ran the Fair Scheduler's core scheduling loop through a profiler tool and identified
a bunch of minimally invasive changes that can shave off a few milliseconds.
> The main one is demoting a couple INFO log messages to DEBUG, which brought my benchmark
down from 16000 ms to 6000.
> A few others (which had way less of an impact) were
> * Most of the time in comparisons was being spent in Math.signum.  I switched this to
direct ifs and elses and it halved the percent of time spent in comparisons.
> * I removed some unnecessary instantiations of Resource objects
> * I made it so that queues' usage wasn't calculated from the applications up each time
getResourceUsage was called.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message