hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Harsh J (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-5785) Derive heap size or mapreduce.*.memory.mb automatically
Date Tue, 03 Nov 2015 10:38:28 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Harsh J updated MAPREDUCE-5785:
    Release Note: 
The memory values for mapreduce.map/reduce.memory.mb keys, if left to their default values
of -1, will now be automatically inferred from the heap size value system property (-Xmx)
specified for mapreduce.map/reduce.java.opts keys.

The converse is also done, i.e. if mapreduce.map/reduce.memory.mb values are specified, but
no -Xmx is supplied for mapreduce.map/reduce.java.opts keys, then the -Xmx value will be derived
from the former's value.

If neither is specified, then a default value of 1024 MB gets used.

For both these conversions, a scaling factor specified by property mapreduce.job.heap.memory-mb.ratio
is used, to account for overheads between heap usage vs. actual physical memory usage.

Existing configs or job code that already specify both the set of properties explicitly would
not be affected by this inferring change.

> Derive heap size or mapreduce.*.memory.mb automatically
> -------------------------------------------------------
>                 Key: MAPREDUCE-5785
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5785
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: mr-am, task
>            Reporter: Gera Shegalov
>            Assignee: Gera Shegalov
>             Fix For: 3.0.0
>         Attachments: MAPREDUCE-5785.v01.patch, MAPREDUCE-5785.v02.patch, MAPREDUCE-5785.v03.patch,
mr-5785-4.patch, mr-5785-5.patch, mr-5785-6.patch, mr-5785-7.patch, mr-5785-8.patch, mr-5785-9.patch
> Currently users have to set 2 memory-related configs per Job / per task type.  One first
chooses some container size map reduce.\*.memory.mb and then a corresponding maximum Java
heap size Xmx < map reduce.\*.memory.mb. This makes sure that the JVM's C-heap (native
memory + Java heap) does not exceed this mapreduce.*.memory.mb. If one forgets to tune Xmx,
MR-AM might be 
> - allocating big containers whereas the JVM will only use the default -Xmx200m.
> - allocating small containers that will OOM because Xmx is too high.
> With this JIRA, we propose to set Xmx automatically based on an empirical ratio that
can be adjusted. Xmx is not changed automatically if provided by the user.

This message was sent by Atlassian JIRA

View raw message