hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ahmed Radwan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4469) Resource calculation in child tasks is CPU-heavy
Date Mon, 06 Aug 2012 20:47:02 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429409#comment-13429409
] 

Ahmed Radwan commented on MAPREDUCE-4469:
-----------------------------------------

Thanks Todd! I think this is a result of the resource calculation done in the task's updateResourceCounters()
which is continuously called to send progress update.

One option is to optionally disable the use of the ResourceCalculatorPlugin in the task, which
will get rid of this overhead. Looking into the code, seems that the ResourceCalculatorPlugin
is only used in the task to update CPU_MILLISECONDS, PHYSICAL_MEMORY_BYTES and VIRTUAL_MEMORY_BYTES
counters which are only referenced in TestTTResourceReporting.
                
> Resource calculation in child tasks is CPU-heavy
> ------------------------------------------------
>
>                 Key: MAPREDUCE-4469
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4469
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: performance, task
>    Affects Versions: 1.0.3
>            Reporter: Todd Lipcon
>
> In doing some benchmarking on a hadoop-1 derived codebase, I noticed that each of the
child tasks was doing a ton of syscalls. Upon stracing, I noticed that it's spending a lot
of time looping through all the files in /proc to calculate resource usage.
> As a test, I added a flag to disable use of the ResourceCalculatorPlugin within the tasks.
On a CPU-bound 500G-sort workload, this improved total job runtime by about 10% (map slot-seconds
by 14%, reduce slot seconds by 8%)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message