Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-issues@hadoop.apache.org
Date: Mon, 24 Nov 2014 22:38:13 +0000 (UTC)
From: "Karthik Kambatla (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Message-ID: <JIRA.12699603.1394282667000.10552.1416868693941@Atlassian.JIRA>
In-Reply-To: <JIRA.12699603.1394282667000@Atlassian.JIRA>
References: <JIRA.12699603.1394282667000@Atlassian.JIRA>
 <JIRA.12699603.1394282667171@arcas>
Subject: [jira] [Commented] (MAPREDUCE-5785) Derive heap size or
 mapreduce.*.memory.mb automatically
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/MAPREDUCE-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14223675#comment-14223675 ] 

Karthik Kambatla commented on MAPREDUCE-5785:
---------------------------------------------

bq. In my patch I was setting DEFAULT_MAPRED_TASK_JAVA_OPTS to "" to make sure that we don't get null. In the latest patch it actually happens and is not caught because the tests are weakened by checking only for containment of command fragments instead of checking the whole command.

Thanks for the input, Gera. You are right. I didn't realize I was weakening the test when I refactored it.  Let me reopen the JIRA and revert the refactoring in an addendum patch. 

> Derive heap size or mapreduce.*.memory.mb automatically
> -------------------------------------------------------
>
>                 Key: MAPREDUCE-5785
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5785
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: mr-am, task
>            Reporter: Gera Shegalov
>            Assignee: Gera Shegalov
>             Fix For: 3.0.0
>
>         Attachments: MAPREDUCE-5785.v01.patch, MAPREDUCE-5785.v02.patch, MAPREDUCE-5785.v03.patch, mr-5785-4.patch, mr-5785-5.patch, mr-5785-6.patch
>
>
> Currently users have to set 2 memory-related configs per Job / per task type.  One first chooses some container size map reduce.\*.memory.mb and then a corresponding maximum Java heap size Xmx < map reduce.\*.memory.mb. This makes sure that the JVM's C-heap (native memory + Java heap) does not exceed this mapreduce.*.memory.mb. If one forgets to tune Xmx, MR-AM might be 
> - allocating big containers whereas the JVM will only use the default -Xmx200m.
> - allocating small containers that will OOM because Xmx is too high.
> With this JIRA, we propose to set Xmx automatically based on an empirical ratio that can be adjusted. Xmx is not changed automatically if provided by the user.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)