hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5785) Derive heap size or mapreduce.*.memory.mb automatically
Date Mon, 24 Nov 2014 22:38:13 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14223675#comment-14223675
] 

Karthik Kambatla commented on MAPREDUCE-5785:
---------------------------------------------

bq. In my patch I was setting DEFAULT_MAPRED_TASK_JAVA_OPTS to "" to make sure that we don't
get null. In the latest patch it actually happens and is not caught because the tests are
weakened by checking only for containment of command fragments instead of checking the whole
command.

Thanks for the input, Gera. You are right. I didn't realize I was weakening the test when
I refactored it.  Let me reopen the JIRA and revert the refactoring in an addendum patch.


> Derive heap size or mapreduce.*.memory.mb automatically
> -------------------------------------------------------
>
>                 Key: MAPREDUCE-5785
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5785
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: mr-am, task
>            Reporter: Gera Shegalov
>            Assignee: Gera Shegalov
>             Fix For: 3.0.0
>
>         Attachments: MAPREDUCE-5785.v01.patch, MAPREDUCE-5785.v02.patch, MAPREDUCE-5785.v03.patch,
mr-5785-4.patch, mr-5785-5.patch, mr-5785-6.patch
>
>
> Currently users have to set 2 memory-related configs per Job / per task type.  One first
chooses some container size map reduce.\*.memory.mb and then a corresponding maximum Java
heap size Xmx < map reduce.\*.memory.mb. This makes sure that the JVM's C-heap (native
memory + Java heap) does not exceed this mapreduce.*.memory.mb. If one forgets to tune Xmx,
MR-AM might be 
> - allocating big containers whereas the JVM will only use the default -Xmx200m.
> - allocating small containers that will OOM because Xmx is too high.
> With this JIRA, we propose to set Xmx automatically based on an empirical ratio that
can be adjusted. Xmx is not changed automatically if provided by the user.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message