hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rajesh Balamohan (JIRA)" <j...@apache.org>
Subject [jira] Created: (MAPREDUCE-1533) reduce or remove usage of String.format() usage in CapacityTaskScheduler.updateQSIObjects
Date Thu, 25 Feb 2010 05:06:27 GMT
reduce or remove usage of String.format() usage in CapacityTaskScheduler.updateQSIObjects
-----------------------------------------------------------------------------------------

                 Key: MAPREDUCE-1533
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1533
             Project: Hadoop Map/Reduce
          Issue Type: Bug
    Affects Versions: 0.20.1
            Reporter: Rajesh Balamohan


When short jobs are executed in hadoop with OutOfBandHeardBeat=true, JT executes heartBeat()
method heavily. This internally makes a call to CapacityTaskScheduler.updateQSIObjects().


CapacityTaskScheduler.updateQSIObjects(), internally calls String.format() for setting the
job scheduling information. Based on the datastructure size of "jobQueuesManager" and "queueInfoMap",
the number of times String.format() gets executed becomes very high. String.format() internally
does pattern matching which turns to be out very heavy (This was revealed while profiling
JT. Almost 57% of time was spent in CapacityScheduler.assignTasks(), out of which String.format()
took 46%.

Would it be possible to do String.format() only at the time of invoking JobInProgress.getSchedulingInfo?.
This might reduce the pressure on JT while processing heartbeats. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message