hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Joseph Evans (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-4760) Make a version of Counters that is composit for the job and stores the counter values in arrays.
Date Wed, 31 Oct 2012 15:13:12 GMT
Robert Joseph Evans created MAPREDUCE-4760:
----------------------------------------------

             Summary: Make a version of Counters that is composit for the job and stores the
counter values in arrays.
                 Key: MAPREDUCE-4760
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4760
             Project: Hadoop Map/Reduce
          Issue Type: Improvement
          Components: mrv2
    Affects Versions: 0.23.4, 2.0.2-alpha
            Reporter: Robert Joseph Evans
            Priority: Minor


String interning reduced the size of counters a lot.  After that and the fix for a memory
leak in the IPC server a job with 20000 map tasks and 3000 reducers takes about 200MB to store
the state of all of the tasks.  Looking at a memory dump of the AM each task attempt has a
pointer to a Counters object that is about 2kb to 3kb in size.  That means Counters account
for about 56MB of the 200MB of state.  This job only had about 40 task counters in it.  Each
counter stores a long value so if we stored them in a long[] instead we should only be taking
up 7MB.

Also assuming that some of the counters only appear in a map task or a reduce task we should
be able to have one CompositCounters for map tasks and one for reduce tasks so it would reduce
the size even further. 

NOTE: without this change I would expect to be able to run a 100,000 task job in the default
1024MB AM heap (875MB/200MB * 2300) I reserved 150MB for IPC buffers and event data.  With
this change we could expect to run about 130,000 tasks (875MB/150MB * 2300).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message