hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Philip Zeyliger (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-220) Collecting cpu and memory usage for MapReduce tasks
Date Wed, 04 Aug 2010 03:23:23 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895129#action_12895129
] 

Philip Zeyliger commented on MAPREDUCE-220:
-------------------------------------------

Hi Scott,

You could also "reset" the counters to 0 when the new task is started (sort of like a "tare"
button on a scale).  If resourceCalculator.getProcCumulativeCpuTime() was rather resourceCalculator.getCumulativeCpuTimeDelta()
[cumulative CPU time since last call], you could use counter.incr() for the CPU usage.

It's also worth mentioning that the memory usage here is the last-known memory usage value.
 It's not byte-seconds (which wouldn't be that useful), nor is it maximum memory.  That seems
useful, but it's a bit unintuitive.

{noformat}
+    long cpuTime = resourceCalculator.getProcCumulativeCpuTime();
+    long pMem = resourceCalculator.getProcPhysicalMemorySize();
+    long vMem = resourceCalculator.getProcVirtualMemorySize();
+    counters.findCounter(TaskCounter.CPU_MILLISECONDS).setValue(cpuTime);
+    counters.findCounter(TaskCounter.PHYSICAL_MEMORY_BYTES).setValue(pMem);
+    counters.findCounter(TaskCounter.VIRTUAL_MEMORY_BYTES).setValue(vMem);
{noformat}

> Collecting cpu and memory usage for MapReduce tasks
> ---------------------------------------------------
>
>                 Key: MAPREDUCE-220
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-220
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: task, tasktracker
>            Reporter: Hong Tang
>            Assignee: Scott Chen
>             Fix For: 0.22.0
>
>         Attachments: MAPREDUCE-220-20100616.txt, MAPREDUCE-220-v1.txt, MAPREDUCE-220.txt
>
>
> It would be nice for TaskTracker to collect cpu and memory usage for individual Map or
Reduce tasks over time.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message