hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Philip Zeyliger (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-220) Collecting cpu and memory usage for MapReduce tasks
Date Wed, 04 Aug 2010 03:23:23 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895129#action_12895129

Philip Zeyliger commented on MAPREDUCE-220:

Hi Scott,

You could also "reset" the counters to 0 when the new task is started (sort of like a "tare"
button on a scale).  If resourceCalculator.getProcCumulativeCpuTime() was rather resourceCalculator.getCumulativeCpuTimeDelta()
[cumulative CPU time since last call], you could use counter.incr() for the CPU usage.

It's also worth mentioning that the memory usage here is the last-known memory usage value.
 It's not byte-seconds (which wouldn't be that useful), nor is it maximum memory.  That seems
useful, but it's a bit unintuitive.

+    long cpuTime = resourceCalculator.getProcCumulativeCpuTime();
+    long pMem = resourceCalculator.getProcPhysicalMemorySize();
+    long vMem = resourceCalculator.getProcVirtualMemorySize();
+    counters.findCounter(TaskCounter.CPU_MILLISECONDS).setValue(cpuTime);
+    counters.findCounter(TaskCounter.PHYSICAL_MEMORY_BYTES).setValue(pMem);
+    counters.findCounter(TaskCounter.VIRTUAL_MEMORY_BYTES).setValue(vMem);

> Collecting cpu and memory usage for MapReduce tasks
> ---------------------------------------------------
>                 Key: MAPREDUCE-220
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-220
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: task, tasktracker
>            Reporter: Hong Tang
>            Assignee: Scott Chen
>             Fix For: 0.22.0
>         Attachments: MAPREDUCE-220-20100616.txt, MAPREDUCE-220-v1.txt, MAPREDUCE-220.txt
> It would be nice for TaskTracker to collect cpu and memory usage for individual Map or
Reduce tasks over time.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message