hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5785) Derive task attempt JVM max heap size automatically from mapreduce.*.memory.mb
Date Mon, 10 Mar 2014 14:45:44 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13925767#comment-13925767

Jason Lowe commented on MAPREDUCE-5785:

Haven't had a chance to look into the patch into great detail, but here are some initial comments:

- should 'memory.mb.xmx.ratio' be 'memory.mb.heap.ratio'?  Even the code names it that internally.
- rather than commenting out the mapred-default property it should leave it in without a value
set.  See the mapred.child.env entry as an example.
- should be easy to add a unit test that verifies the ratio is working as intended, e.g.:
changing it sees a corresponding jvm argument change out of MapReduceChildJVM.getVMCommand
and setting an explicit heap setting in the config prevents the ratio from taking effect.

> Derive task attempt JVM max heap size automatically from mapreduce.*.memory.mb
> ------------------------------------------------------------------------------
>                 Key: MAPREDUCE-5785
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5785
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: mr-am, task
>            Reporter: Gera Shegalov
>            Assignee: Gera Shegalov
>         Attachments: MAPREDUCE-5785.v01.patch
> Currently users have to set 2 memory-related configs per Job / per task type.  One first
chooses some container size map reduce.\*.memory.mb and then a corresponding maximum Java
heap size Xmx < map reduce.\*.memory.mb. This makes sure that the JVM's C-heap (native
memory + Java heap) does not exceed this mapreduce.*.memory.mb. If one forgets to tune Xmx,
MR-AM might be 
> - allocating big containers whereas the JVM will only use the default -Xmx200m.
> - allocating small containers that will OOM because Xmx is too high.
> With this JIRA, we propose to set Xmx automatically based on an empirical ratio that
can be adjusted. Xmx is not changed automatically if provided by the user.

This message was sent by Atlassian JIRA

View raw message