hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Boris Ryakhovskiy (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-4715) Heap memory growing when JVM reuse option enabled
Date Tue, 09 Oct 2012 14:13:03 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-4715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Boris Ryakhovskiy updated MAPREDUCE-4715:

    Attachment: VisualVM_screenshot.png

VisualVM screenshot attached
> Heap memory growing when JVM reuse option enabled
> -------------------------------------------------
>                 Key: MAPREDUCE-4715
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4715
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: job submission
>    Affects Versions: 0.20.2
>         Environment: Linux, cdh3u5
>            Reporter: Boris Ryakhovskiy
>            Priority: Minor
>         Attachments: VisualVM_screenshot.png
> When mapred.job.reuse.jvm.num.tasks option is set to 100 or more tasks, JVM fails with
"java.lang.OutOfMemoryError: Java heap space" error message. Jobs itself are small and job
heap size is set to -Xmx200m .
> It looks like the reason of the issue is a FileSystem$Cache object which collects JobConf
objects from all jobs which were executed in current JVM. GC root for the FileSystem$Cache
is ApplicationShutdownHooks class (cannot attach screenshot from visualvm)
> In my case each of JobConf objects allocates ~500K (50M total in case of 100 jobs).

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message