Return-Path:
The child-task inherits the environment of the parent
null
.
- * @param memoryLimit virtual memory limit
- * @return a String[]
with the ulimit command arguments or
- * null
if we are running on a non *nix platform or
- * if the limit is unspecified.
- */
- public static String[] getUlimitMemoryCommand(int memoryLimit) {
- // ulimit isn't supported on Windows
- if (WINDOWS) {
- return null;
- }
-
- return new String[] {ULIMIT_COMMAND, "-v", String.valueOf(memoryLimit)};
- }
-
/**
* Get the Unix command for setting the maximum virtual memory available
* to a given child process. This is only relevant when we are forking a
@@ -107,9 +82,7 @@ abstract public class Shell {
* @return a String[]
with the ulimit command arguments or
* null
if we are running on a non *nix platform or
* if the limit is unspecified.
- * @deprecated Use {@link #getUlimitMemoryCommand(int)}
*/
- @Deprecated
public static String[] getUlimitMemoryCommand(Configuration conf) {
// ulimit isn't supported on Windows
if (WINDOWS) {
@@ -124,8 +97,8 @@ abstract public class Shell {
// Parse it to ensure it is legal/sane
int memoryLimit = Integer.valueOf(ulimit);
-
- return getUlimitMemoryCommand(memoryLimit);
+
+ return new String[] {"ulimit", "-v", String.valueOf(memoryLimit)};
}
/** Set to true on Windows platforms */
Modified: hadoop/common/branches/branch-0.20-security-patches/src/docs/src/documentation/content/xdocs/cluster_setup.xml
URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20-security-patches/src/docs/src/documentation/content/xdocs/cluster_setup.xml?rev=1077105&r1=1077104&r2=1077105&view=diff
==============================================================================
--- hadoop/common/branches/branch-0.20-security-patches/src/docs/src/documentation/content/xdocs/cluster_setup.xml (original)
+++ hadoop/common/branches/branch-0.20-security-patches/src/docs/src/documentation/content/xdocs/cluster_setup.xml Fri Mar 4 03:41:02 2011
@@ -392,18 +392,10 @@
- conf/mapred-site.xml
- mapred.map.child.java.opts
+ mapred.child.java.opts
-Xmx512M
- Larger heap-size for child jvms of maps.
-
-
-
conf/mapred-site.xml
- mapred.reduce.child.java.opts
- -Xmx512M
-
- Larger heap-size for child jvms of reduces.
+ Larger heap-size for child jvms of maps/reduces.
@@ -473,17 +465,9 @@
- conf/mapred-site.xml
- mapred.map.child.java.opts
- -Xmx512M
-
- Larger heap-size for child jvms of maps.
-
-
-
Modified: hadoop/common/branches/branch-0.20-security-patches/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml
URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20-security-patches/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml?rev=1077105&r1=1077104&r2=1077105&view=diff
==============================================================================
--- hadoop/common/branches/branch-0.20-security-patches/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml (original)
+++ hadoop/common/branches/branch-0.20-security-patches/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml Fri Mar 4 03:41:02 2011
@@ -1051,25 +1051,24 @@
conf/mapred-site.xml
- mapred.reduce.child.java.opts
+ mapred.child.java.opts
-Xmx1024M
- Larger heap-size for child jvms of reduces.
+ Larger heap-size for child jvms of maps/reduces.
TaskTracker
. The user can specify additional options to the
- child-jvm via the mapred.{map|reduce}.child.java.opts
- configuration parameter in the JobConf
such as non-standard
- paths for the run-time linker to search shared libraries via
+ child-jvm via the mapred.child.java.opts
configuration
+ parameter in the JobConf
such as non-standard paths for the
+ run-time linker to search shared libraries via
-Djava.library.path=<>
etc. If the
- mapred.{map|reduce}.child.java.opts
parameters contains the
- symbol @taskid@ it is interpolated with value of
- taskid
of the map/reduce task.mapred.child.java.opts
contains the symbol @taskid@
+ it is interpolated with value of taskid
of the map/reduce
+ task.
Here is an example with multiple arguments and substitutions,
showing jvm GC logging, and start of a passwordless JVM JMX agent so that
it can connect with jconsole and the likes to watch child memory,
threads and get thread dumps. It also sets the maximum heap-size of the
- map and reduce child jvm to 512MB & 1024MB respectively. It also
- adds an additional path to the java.library.path
of the
- child-jvm.
java.library.path
of the child-jvm.
<property>
- <name>mapred.map.child.java.opts</name>
+ <name>mapred.child.java.opts</name>
<value>
-Xmx512M -Djava.library.path=/home/mycompany/lib
@@ -1081,33 +1080,19 @@
</property>
- <property>
- <name>mapred.reduce.child.java.opts</name>
- <value>
-
- -Xmx1024M -Djava.library.path=/home/mycompany/lib
- -verbose:gc -Xloggc:/tmp/@taskid@.gc
-
- -Dcom.sun.management.jmxremote.authenticate=false
- -Dcom.sun.management.jmxremote.ssl=false
- </value>
- </property>
-
Users/admins can also specify the maximum virtual memory
of the launched child-task, and any sub-process it launches
- recursively, using mapred.{map|reduce}.child.ulimit
. Note
- that the value set here is a per process limit.
- The value for mapred.{map|reduce}.child.ulimit
should be
- specified in kilo bytes (KB). And also the value must be greater than
+ recursively, using mapred.child.ulimit
. Note that
+ the value set here is a per process limit.
+ The value for mapred.child.ulimit
should be specified
+ in kilo bytes (KB). And also the value must be greater than
or equal to the -Xmx passed to JavaVM, else the VM might not start.
Note: mapred.{map|reduce}.child.java.opts
are used only
- for configuring the launched child tasks from task tracker. Configuring
+
Note: mapred.child.java.opts
are used only for
+ configuring the launched child tasks from task tracker. Configuring
the memory options for daemons is documented in
cluster_setup.html
mapred.reduce.child.java.opts
-
+ as typically specified in mapred.child.java.opts
-
that can be allocated to storing map outputs during the
shuffle. Though some memory should be set aside for the
framework, in general it is advantageous to set this high
Modified: hadoop/common/branches/branch-0.20-security-patches/src/mapred/org/apache/hadoop/mapred/JobConf.java
URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20-security-patches/src/mapred/org/apache/hadoop/mapred/JobConf.java?rev=1077105&r1=1077104&r2=1077105&view=diff
==============================================================================
--- hadoop/common/branches/branch-0.20-security-patches/src/mapred/org/apache/hadoop/mapred/JobConf.java (original)
+++ hadoop/common/branches/branch-0.20-security-patches/src/mapred/org/apache/hadoop/mapred/JobConf.java Fri Mar 4 03:41:02 2011
@@ -159,153 +159,6 @@ public class JobConf extends Configurati
"mapred.job.reduce.memory.mb";
/**
- * Configuration key to set the java command line options for the child
- * map and reduce tasks.
- *
- * Java opts for the task tracker child processes.
- * The following symbol, if present, will be interpolated: @taskid@.
- * It is replaced by current TaskID. Any other occurrences of '@' will go
- * unchanged.
- * For example, to enable verbose gc logging to a file named for the taskid in
- * /tmp and to set the heap maximum to be a gigabyte, pass a 'value' of:
- * -Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc
- *
- * The configuration variable {@link #MAPRED_TASK_ULIMIT} can be used to
- * control the maximum virtual memory of the child processes.
- *
- * The configuration variable {@link #MAPRED_TASK_ENV} can be used to pass
- * other environment variables to the child processes.
- *
- * @deprecated Use {@link #MAPRED_MAP_TASK_JAVA_OPTS} or
- * {@link #MAPRED_REDUCE_TASK_JAVA_OPTS}
- */
- @Deprecated
- public static final String MAPRED_TASK_JAVA_OPTS = "mapred.child.java.opts";
-
- /**
- * Configuration key to set the java command line options for the map tasks.
- *
- * Java opts for the task tracker child map processes.
- * The following symbol, if present, will be interpolated: @taskid@.
- * It is replaced by current TaskID. Any other occurrences of '@' will go
- * unchanged.
- * For example, to enable verbose gc logging to a file named for the taskid in
- * /tmp and to set the heap maximum to be a gigabyte, pass a 'value' of:
- * -Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc
- *
- * The configuration variable {@link #MAPRED_MAP_TASK_ULIMIT} can be used to
- * control the maximum virtual memory of the map processes.
- *
- * The configuration variable {@link #MAPRED_MAP_TASK_ENV} can be used to pass
- * other environment variables to the map processes.
- */
- public static final String MAPRED_MAP_TASK_JAVA_OPTS =
- "mapred.map.child.java.opts";
-
- /**
- * Configuration key to set the java command line options for the reduce tasks.
- *
- * Java opts for the task tracker child reduce processes.
- * The following symbol, if present, will be interpolated: @taskid@.
- * It is replaced by current TaskID. Any other occurrences of '@' will go
- * unchanged.
- * For example, to enable verbose gc logging to a file named for the taskid in
- * /tmp and to set the heap maximum to be a gigabyte, pass a 'value' of:
- * -Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc
- *
- * The configuration variable {@link #MAPRED_REDUCE_TASK_ULIMIT} can be used
- * to control the maximum virtual memory of the reduce processes.
- *
- * The configuration variable {@link #MAPRED_REDUCE_TASK_ENV} can be used to
- * pass process environment variables to the reduce processes.
- */
- public static final String MAPRED_REDUCE_TASK_JAVA_OPTS =
- "mapred.reduce.child.java.opts";
-
- public static final String DEFAULT_MAPRED_TASK_JAVA_OPTS = "-Xmx200m";
-
- /**
- * Configuration key to set the maximum virutal memory available to the child
- * map and reduce tasks (in kilo-bytes).
- *
- * Note: This must be greater than or equal to the -Xmx passed to the JavaVM
- * via {@link #MAPRED_TASK_JAVA_OPTS}, else the VM might not start.
- *
- * @deprecated Use {@link #MAPRED_MAP_TASK_ULIMIT} or
- * {@link #MAPRED_REDUCE_TASK_ULIMIT}
- */
- @Deprecated
- public static final String MAPRED_TASK_ULIMIT = "mapred.child.ulimit";
-
- /**
- * Configuration key to set the maximum virutal memory available to the
- * map tasks (in kilo-bytes).
- *
- * Note: This must be greater than or equal to the -Xmx passed to the JavaVM
- * via {@link #MAPRED_MAP_TASK_JAVA_OPTS}, else the VM might not start.
- */
- public static final String MAPRED_MAP_TASK_ULIMIT = "mapred.map.child.ulimit";
-
- /**
- * Configuration key to set the maximum virutal memory available to the
- * reduce tasks (in kilo-bytes).
- *
- * Note: This must be greater than or equal to the -Xmx passed to the JavaVM
- * via {@link #MAPRED_REDUCE_TASK_JAVA_OPTS}, else the VM might not start.
- */
- public static final String MAPRED_REDUCE_TASK_ULIMIT =
- "mapred.reduce.child.ulimit";
-
- /**
- * Configuration key to set the environment of the child map/reduce tasks.
- *
- * The format of the value is k1=v1,k2=v2
. Further it can
- * reference existing environment variables via $key
.
- *
- * Example:
- * k1=v1,k2=v2
. Further it can
- * reference existing environment variables via $key
.
- *
- * Example:
- * k1=v1,k2=v2
. Further it can
- * reference existing environment variables via $key
.
- *
- * Example:
- * -1
if
- * none is specified
- * @deprecated Use limits specific to the map or reduce tasks set via
- * {@link JobConf#MAPRED_MAP_TASK_ULIMIT} or
- * {@link JobConf#MAPRED_REDUCE_TASK_ULIMIT}
- */
- @Deprecated
- public int getChildUlimit(JobConf jobConf) {
- return jobConf.getInt(JobConf.MAPRED_TASK_ULIMIT, -1);
- }
-
- /**
- * Get the environment variables for the child map/reduce tasks.
- * @param jobConf job configuration
- * @return the environment variables for the child map/reduce tasks or
- * null
if unspecified
- * @deprecated Use environment variables specific to the map or reduce tasks
- * set via {@link JobConf#MAPRED_MAP_TASK_ENV} or
- * {@link JobConf#MAPRED_REDUCE_TASK_ENV}
- */
- public String getChildEnv(JobConf jobConf) {
- return jobConf.get(JobConf.MAPRED_TASK_ENV);
- }
-
@Override
public final void run() {
String errorInfo = "Child Error";
@@ -315,8 +274,8 @@ abstract class TaskRunner extends Thread
// Add child (task) java-vm options.
//
- // The following symbols if present in mapred.{map|reduce}.child.java.opts
- // value are replaced:
+ // The following symbols if present in mapred.child.java.opts value are
+ // replaced:
// + @taskid@ is interpolated with value of TaskID.
// Other occurrences of @ will not be altered.
//
@@ -326,23 +285,14 @@ abstract class TaskRunner extends Thread
// and get thread dumps.
//
//