hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amar Kamat (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2469) Task counters should also report the total heap usage of the task
Date Tue, 17 May 2011 10:23:47 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034681#comment-13034681
] 

Amar Kamat commented on MAPREDUCE-2469:
---------------------------------------

The patch looks fine to me. For the test case, can we do the following
1. Start a Mini-MR cluster.
2. Submit a no-op job (say job1) with 1 map and 1 reduce
3. Submit a word count like job (say job2) with 1 map and 1 reduce.
4. Check if job1's heap usage (reported via its counters) is less than that of job2.

For (3), any job that can add some extra load in the task's heap should suffice. Also both
the jobs should have 'mapred.child.java.opts' having -Xms set to a very low value.

Thoughts?

> Task counters should also report the total heap usage of the task
> -----------------------------------------------------------------
>
>                 Key: MAPREDUCE-2469
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2469
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: task
>    Affects Versions: 0.23.0
>            Reporter: Amar Kamat
>            Assignee: Ravi Gummadi
>              Labels: mapreduce
>         Attachments: 2469.v0.1.patch
>
>
> Currently, the task counters report VSS and RSS usage of the task. The task counter should
also report the total heap usage of the task also. The task might be configured with a max
heap size of M but the task's total heap usage might only be H, where H < M. In such a
case, knowing only M doesn't provide a complete picture of the task's memory usage.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message