hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Kimball (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-3) Set mapred.child.ulimit automatically to the value of the RAM limits for a job, if they are set
Date Sat, 11 Jul 2009 00:53:14 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729916#action_12729916
] 

Aaron Kimball commented on MAPREDUCE-3:
---------------------------------------

A caution here is that {{mapred.child.ulimit}} needs to account for the memory overhead of
the JVM itself. Merely setting {{mapred.child.ulimit}} to the same value as the {{-Xmx}}_hhhh_{{m}}
in {{mapred.child.java.opts}} will fail to launch child tasks. You'll need some overhead room;
I don't know exactly how much.



> Set mapred.child.ulimit automatically to the value of the RAM limits for a job, if they
are set
> -----------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-3
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Hemanth Yamijala
>
> Memory based monitoring and scheduling allow users to set memory limits for the tasks
of their jobs. This parameter is the total memory taken by the task, and any children it may
launch (for e.g. in the case of streaming). A related parameter is mapred.child.ulimit which
is a hard limit on the memory used by a single process of the entire task tree. For user convenience,
it would be sensible for the system to set the ulimit to atleast the memory required by the
task, if the user has specified the latter.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message