Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-issues@hadoop.apache.org
Message-ID: <4577296.92031283256654803.JavaMail.jira@thor>
Date: Tue, 31 Aug 2010 08:10:54 -0400 (EDT)
From: "Liyin Liang (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Subject: [jira] Commented: (MAPREDUCE-1533) Reduce or remove usage of
 String.format() usage in CapacityTaskScheduler.updateQSIObjects and
 Counters.makeEscapedString()
In-Reply-To: <249435897.516781267074387969.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/MAPREDUCE-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904607#action_12904607 ] 

Liyin Liang commented on MAPREDUCE-1533:
----------------------------------------

The patch has a small problem.
I think line257-259 of Counters.java should be:
for(String subcounter: subcountersArray) {
  builder.append(subcounter);
}

instead of:
for(Counter counter: subcounters.values()) {
  builder.append(counter.makeEscapedCompactString());
}

> Reduce or remove usage of String.format() usage in CapacityTaskScheduler.updateQSIObjects and Counters.makeEscapedString()
> --------------------------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1533
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1533
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: jobtracker
>    Affects Versions: 0.20.1
>            Reporter: Rajesh Balamohan
>            Assignee: Dick King
>             Fix For: 0.22.0
>
>         Attachments: mapreduce-1533--2010-05-10a.patch, mapreduce-1533--2010-05-21.patch, mapreduce-1533--2010-05-21a.patch, mapreduce-1533--2010-05-24.patch, MAPREDUCE-1533-and-others-20100413.1.txt, MAPREDUCE-1533-and-others-20100413.bugfix.txt, mapreduce-1533-v1.4.patch, mapreduce-1533-v1.8.patch
>
>
> When short jobs are executed in hadoop with OutOfBandHeardBeat=true, JT executes heartBeat() method heavily. This internally makes a call to CapacityTaskScheduler.updateQSIObjects(). 
> CapacityTaskScheduler.updateQSIObjects(), internally calls String.format() for setting the job scheduling information. Based on the datastructure size of "jobQueuesManager" and "queueInfoMap", the number of times String.format() gets executed becomes very high. String.format() internally does pattern matching which turns to be out very heavy (This was revealed while profiling JT. Almost 57% of time was spent in CapacityScheduler.assignTasks(), out of which String.format() took 46%.
> Would it be possible to do String.format() only at the time of invoking JobInProgress.getSchedulingInfo?. This might reduce the pressure on JT while processing heartbeats. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.