hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron T. Myers (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-4760) Make a version of Counters that is composite for the job and stores the counter values in arrays
Date Wed, 31 Oct 2012 17:40:12 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-4760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Aaron T. Myers updated MAPREDUCE-4760:
--------------------------------------

    Summary: Make a version of Counters that is composite for the job and stores the counter
values in arrays  (was: Make a version of Counters that is composit for the job and stores
the counter values in arrays.)
    
> Make a version of Counters that is composite for the job and stores the counter values
in arrays
> ------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-4760
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4760
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mrv2
>    Affects Versions: 2.0.2-alpha, 0.23.4
>            Reporter: Robert Joseph Evans
>            Priority: Minor
>
> String interning reduced the size of counters a lot.  After that and the fix for a memory
leak in the IPC server a job with 20000 map tasks and 3000 reducers takes about 200MB to store
the state of all of the tasks.  Looking at a memory dump of the AM each task attempt has a
pointer to a Counters object that is about 2kb to 3kb in size.  That means Counters account
for about 56MB of the 200MB of state.  This job only had about 40 task counters in it.  Each
counter stores a long value so if we stored them in a long[] instead we should only be taking
up 7MB.
> Also assuming that some of the counters only appear in a map task or a reduce task we
should be able to have one CompositCounters for map tasks and one for reduce tasks so it would
reduce the size even further. 
> NOTE: without this change I would expect to be able to run a 100,000 task job in the
default 1024MB AM heap (875MB/200MB * 2300) I reserved 150MB for IPC buffers and event data.
 With this change we could expect to run about 130,000 tasks (875MB/150MB * 2300).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message