hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Chen (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-220) Collecting cpu and memory usage for MapReduce tasks
Date Fri, 18 Jun 2010 00:31:31 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880017#action_12880017

Scott Chen commented on MAPREDUCE-220:

The failed contrib test is TestSimulatorDeterministicReplay.testMain.
It is a know issue in MAPREDUCE-1834.

In the patch we put task cumulative CPU time, current physical memory and current virtual
memory in task counters.
So it will be aggregated in JobInProgress.getJobCounter(). We will get the total CPU time
and current total memory usage.
They will go to both web ui and history as part of the counters.
We can access the task counters to obtain these information in place like task scheduler too.

@Vinod: I didn't do the refactoring of the ProcTree. Because LinuxResourceCalculatorPlugin
is now called by the task (updateCounters is in Task.java). It is in a different process than
TaskTracker so we can not reuse the ProcTree. But directory /proc/ is in memory, this may
not be so bad in terms of performance. What do you think?

> Collecting cpu and memory usage for MapReduce tasks
> ---------------------------------------------------
>                 Key: MAPREDUCE-220
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-220
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: task, tasktracker
>            Reporter: Hong Tang
>            Assignee: Scott Chen
>             Fix For: 0.22.0
>         Attachments: MAPREDUCE-220-20100616.txt, MAPREDUCE-220-v1.txt, MAPREDUCE-220.txt
> It would be nice for TaskTracker to collect cpu and memory usage for individual Map or
Reduce tasks over time.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message