hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-2345) Optimize jobtracker's memory usage
Date Mon, 28 Feb 2011 18:39:36 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000457#comment-13000457
] 

Arun C Murthy commented on MAPREDUCE-2345:
------------------------------------------

Meng, interesting analysis, thanks!

To be perfectly honest, I'm surprised you guys are seeing this many *memory* issues with the
JT... what version of the Hadoop Map-Reduce are you running? A simple solution we have deployed
at Yahoo! for a long while now is to aggressively cut down #completed jobs in memory which
has helped a *lot*. Something to consider for you guys.

> Optimize jobtracker's  memory usage  
> -------------------------------------
>
>                 Key: MAPREDUCE-2345
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2345
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker
>    Affects Versions: 0.21.0
>            Reporter: MengWang
>              Labels: hadoop
>             Fix For: 0.23.0
>
>         Attachments: jt-memory-useage.bmp
>
>
> To many tasks will eat up a considerable amount of JobTracker's heap space. According
to our observation, 50GB heap size can support to 5,000,000 tasks, so we should optimize jobtracker's
memory usage for more jobs and tasks. Yourkit java profile show that counters, duplicate strings,
Task waste too much memory. Our optimization around these three points reduced jobtracker's
memory to 1/3. 

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message