Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-issues@hadoop.apache.org
Date: Mon, 10 Mar 2014 12:35:43 +0000 (UTC)
From: "Gera Shegalov (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Message-ID: <JIRA.12699603.1394282667171.35114.1394454943253@arcas>
In-Reply-To: <JIRA.12699603.1394282667171@arcas>
References: <JIRA.12699603.1394282667171@arcas>
Subject: [jira] [Updated] (MAPREDUCE-5785) Derive task attempt JVM max heap
 size automatically from mapreduce.*.memory.mb
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/MAPREDUCE-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gera Shegalov updated MAPREDUCE-5785:
-------------------------------------

    Description: 
Currently users have to set 2 memory-related configs per Job / per task type.  One first chooses some container size mapreduce.*.memory.mb and then a corresponding maximum Java heap size Xmx < mapreduce.*.memory.mb. This makes sure that the JVM's C-heap (native memory + Java heap) does not exceed this mapreduce.*.memory.mb. If one forgets to tune Xmx, MR-AM might be 
- allocating big containers whereas the JVM will only use the default -Xmx200m.
- allocating small containers that will OOM because Xmx is too high.

With this JIRA, we propose to set Xmx automatically based on an empirical ratio that can be adjusted. Xmx is not changed automatically if provided by the user.


  was:
Currently users have to set 2 memory-related configs for per Job / per task type.  One fist choses some container size mapreduce.*.memory.mb and then a corresponding Xmx < mapreduce.*.memory.mb to make sure that the JVM with the user code heap, and its native memory do not  exceed this limit. If one forgets to tune Xmx, MR-AM might be allocating big containers whereas the JVM will only use the default -Xmx200m.

With this JIRA, we propose to set Xmx automatically base on an empirical ratio that can be adjusted. Xmx is not changed automaically if provided by the user.


> Derive task attempt JVM max heap size automatically from mapreduce.*.memory.mb
> ------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5785
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5785
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: mr-am, task
>            Reporter: Gera Shegalov
>            Assignee: Gera Shegalov
>         Attachments: MAPREDUCE-5785.v01.patch
>
>
> Currently users have to set 2 memory-related configs per Job / per task type.  One first chooses some container size mapreduce.*.memory.mb and then a corresponding maximum Java heap size Xmx < mapreduce.*.memory.mb. This makes sure that the JVM's C-heap (native memory + Java heap) does not exceed this mapreduce.*.memory.mb. If one forgets to tune Xmx, MR-AM might be 
> - allocating big containers whereas the JVM will only use the default -Xmx200m.
> - allocating small containers that will OOM because Xmx is too high.
> With this JIRA, we propose to set Xmx automatically based on an empirical ratio that can be adjusted. Xmx is not changed automatically if provided by the user.


--
This message was sent by Atlassian JIRA
(v6.2#6252)