hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gera Shegalov (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-5785) Derive task attempt JVM max heap size automatically from mapreduce.*.memory.mb
Date Sat, 08 Mar 2014 12:46:43 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Gera Shegalov updated MAPREDUCE-5785:
-------------------------------------

    Description: 
Currently users have to set 2 memory-related configs for per Job / per task type.  One fist
choses some container size mapreduce.*.memory.mb and then a corresponding Xmx < mapreduce.*.memory.mb
to make sure that the JVM with the user code heap, and its native memory do not  exceed this
limit. If one forgets to tune Xmx, MR-AM might be allocating big containers whereas the JVM
will only use the default -Xmx200m.

With this JIRA, we propose to set Xmx automatically base on an empirical ratio that can be
adjusted. Xmx is not changed automaically if provided by the user.


  was:
Currently users have to set 2 memory-related configs for per Job / per task type.  One fist
choses some container size mapreduce.*.memory.mb and then a corresponding Xmx < mapreduce.*.memory.mb
to make sure that the JVM with the user code heap, and its native memory do not  exceed this
limit. If one forgets to tune Xmx, MR-AM might be allocating big containers whereas the JVM
will only use the default -Xmx200m.

With this JIRA, we propose to set Xmx automatically base on an empirical ratio that can be
adjusted. Xmx is not changed if provided by the user.



> Derive task attempt JVM max heap size automatically from mapreduce.*.memory.mb
> ------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5785
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5785
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: mr-am, task
>            Reporter: Gera Shegalov
>            Assignee: Gera Shegalov
>
> Currently users have to set 2 memory-related configs for per Job / per task type.  One
fist choses some container size mapreduce.*.memory.mb and then a corresponding Xmx < mapreduce.*.memory.mb
to make sure that the JVM with the user code heap, and its native memory do not  exceed this
limit. If one forgets to tune Xmx, MR-AM might be allocating big containers whereas the JVM
will only use the default -Xmx200m.
> With this JIRA, we propose to set Xmx automatically base on an empirical ratio that can
be adjusted. Xmx is not changed automaically if provided by the user.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message