hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1048) Sort500 failing since counters patch went in
Date Tue, 27 Feb 2007 20:47:05 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12476375
] 

Doug Cutting commented on HADOOP-1048:
--------------------------------------

Another possible optimization in JobInProgress#updateTaskStatus might be to keep, for each
task, its previous counter values, then compute the difference between those and the new values,
and update global counters incrementally rather than recomputing them from scratch each time.
 This might not actually reduce the amount of computation much, but it might greatly decrease
the amount of allocated objects, and object allocations are considerably more expensive than
arithmetic.

> Sort500 failing since counters patch went in
> --------------------------------------------
>
>                 Key: HADOOP-1048
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1048
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>            Reporter: David Bowen
>
> Nigel wrote:
>   We have a problem
>   Looks like counters patch is "breaking" things
>   Sort500 has been failing since the patch went in
>   Looking at the JT logs, there are 0 "Call queue overflow" message before the patch
went in and 15000+ after the patch went in
>   Looks like the counters stuff is overwhelming the JT 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message