hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-3567) Extraneous JobConf objects in AM heap
Date Thu, 22 Dec 2011 20:13:30 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-3567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Vinod Kumar Vavilapalli updated MAPREDUCE-3567:

    Attachment: MAPREDUCE-3567-20111222.txt

Some of the javac warnings are bogus, they are related to JobConf deprecation and I already
suppressed them. Addresses a couple of warnings which are valid.
> Extraneous JobConf objects in AM heap
> -------------------------------------
>                 Key: MAPREDUCE-3567
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3567
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: mr-am, mrv2, performance
>    Affects Versions: 0.23.0
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Vinod Kumar Vavilapalli
>             Fix For: 0.23.1
>         Attachments: MAPREDUCE-3567-20111215.1.txt, MAPREDUCE-3567-20111222.txt
> MR AM creates new JobConf objects unnecessarily in a couple of places in JobImpl and
TaskImpl which occupy non-trivial amount of heap.
> While working with a 64 bit JVM on 100K maps jobs, with uncompressed pointers, removing
those extraneous objects helped in addressing OOM with 2GB AM heap size.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message