Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-issues@hadoop.apache.org
Date: Mon, 10 Mar 2014 14:45:44 +0000 (UTC)
From: "Jason Lowe (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Message-ID: <JIRA.12699603.1394282667171.35488.1394462744209@arcas>
In-Reply-To: <JIRA.12699603.1394282667171@arcas>
References: <JIRA.12699603.1394282667171@arcas>
Subject: [jira] [Commented] (MAPREDUCE-5785) Derive task attempt JVM max
 heap size automatically from mapreduce.*.memory.mb
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/MAPREDUCE-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13925767#comment-13925767 ] 

Jason Lowe commented on MAPREDUCE-5785:
---------------------------------------

Haven't had a chance to look into the patch into great detail, but here are some initial comments:

- should 'memory.mb.xmx.ratio' be 'memory.mb.heap.ratio'?  Even the code names it that internally. ;-)
- rather than commenting out the mapred-default property it should leave it in without a value set.  See the mapred.child.env entry as an example.
- should be easy to add a unit test that verifies the ratio is working as intended, e.g.: changing it sees a corresponding jvm argument change out of MapReduceChildJVM.getVMCommand and setting an explicit heap setting in the config prevents the ratio from taking effect.

> Derive task attempt JVM max heap size automatically from mapreduce.*.memory.mb
> ------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5785
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5785
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: mr-am, task
>            Reporter: Gera Shegalov
>            Assignee: Gera Shegalov
>         Attachments: MAPREDUCE-5785.v01.patch
>
>
> Currently users have to set 2 memory-related configs per Job / per task type.  One first chooses some container size map reduce.\*.memory.mb and then a corresponding maximum Java heap size Xmx < map reduce.\*.memory.mb. This makes sure that the JVM's C-heap (native memory + Java heap) does not exceed this mapreduce.*.memory.mb. If one forgets to tune Xmx, MR-AM might be 
> - allocating big containers whereas the JVM will only use the default -Xmx200m.
> - allocating small containers that will OOM because Xmx is too high.
> With this JIRA, we propose to set Xmx automatically based on an empirical ratio that can be adjusted. Xmx is not changed automatically if provided by the user.


--
This message was sent by Atlassian JIRA
(v6.2#6252)