hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sandy Ryza (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-5578) Miscellaneous Fair Scheduler speedups
Date Fri, 11 Oct 2013 07:38:42 GMT
Sandy Ryza created MAPREDUCE-5578:

             Summary: Miscellaneous Fair Scheduler speedups
                 Key: MAPREDUCE-5578
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5578
             Project: Hadoop Map/Reduce
          Issue Type: Improvement
          Components: scheduler
            Reporter: Sandy Ryza
            Assignee: Sandy Ryza

I ran the Fair Scheduler's core scheduling loop through a profiler to and identified a bunch
of minimally invasive changes that can shave off a few milliseconds.

The main one is demoting a couple INFO log messages to DEBUG, which brought my benchmark down
from 16000 ms to 6000.

A few others (which had way less of an impact) were
* Most of the time in comparisons was being spent in Math.signum.  I switched this to direct
ifs and elses and it halved the percent of time spent in comparisons.
* I removed some unnecessary instantiations of Resource objects
* I made it so that queues' usage wasn't calculated from the applications up each time getResourceUsage
was called.

This message was sent by Atlassian JIRA

View raw message