hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-3567) Extraneous JobConf objects in AM heap
Date Wed, 21 Dec 2011 00:27:31 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-3567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Vinod Kumar Vavilapalli updated MAPREDUCE-3567:
-----------------------------------------------

    Status: Patch Available  (was: Open)

Gone through the patch again, nothing more to cleanup. Not possible to write new tests as
we are simply replacing references to configuration with JobConf.

Existing tests should continue to validate this changed code.
                
> Extraneous JobConf objects in AM heap
> -------------------------------------
>
>                 Key: MAPREDUCE-3567
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3567
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: mr-am, mrv2, performance
>    Affects Versions: 0.23.0
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Vinod Kumar Vavilapalli
>             Fix For: 0.23.1
>
>         Attachments: MAPREDUCE-3567-20111215.1.txt
>
>
> MR AM creates new JobConf objects unnecessarily in a couple of places in JobImpl and
TaskImpl which occupy non-trivial amount of heap.
> While working with a 64 bit JVM on 100K maps jobs, with uncompressed pointers, removing
those extraneous objects helped in addressing OOM with 2GB AM heap size.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message