hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sandy Ryza (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5785) Derive task attempt JVM max heap size automatically from mapreduce.*.memory.mb
Date Sun, 09 Mar 2014 09:33:43 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13925159#comment-13925159

Sandy Ryza commented on MAPREDUCE-5785:

Something like this has been long-needed.

Though I'm worried that it's not backwards-compatible - users would see their JVM max heaps
change in certain situations.  In situations where they didn't set the max heap, were cutting
it close, but were still OK, they could see OutOfMemoryErrors after the change.

Another thing is that, as a user, I care more about my max heap size than how much I request
from YARN.  The latter is usually a consequence of the former.

One possible way around both of these would be to add a new parameter that controls max heap
size and sets mapreduce.*.memory.mb accordingly.

> Derive task attempt JVM max heap size automatically from mapreduce.*.memory.mb
> ------------------------------------------------------------------------------
>                 Key: MAPREDUCE-5785
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5785
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: mr-am, task
>            Reporter: Gera Shegalov
>            Assignee: Gera Shegalov
>         Attachments: MAPREDUCE-5785.v01.patch
> Currently users have to set 2 memory-related configs for per Job / per task type.  One
fist choses some container size mapreduce.*.memory.mb and then a corresponding Xmx < mapreduce.*.memory.mb
to make sure that the JVM with the user code heap, and its native memory do not  exceed this
limit. If one forgets to tune Xmx, MR-AM might be allocating big containers whereas the JVM
will only use the default -Xmx200m.
> With this JIRA, we propose to set Xmx automatically base on an empirical ratio that can
be adjusted. Xmx is not changed automaically if provided by the user.

This message was sent by Atlassian JIRA

View raw message