hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tomwh...@apache.org
Subject svn commit: r1450835 - in /hadoop/common/branches/branch-1: CHANGES.txt src/docs/src/documentation/content/xdocs/cluster_setup.xml
Date Wed, 27 Feb 2013 16:37:54 GMT
Author: tomwhite
Date: Wed Feb 27 16:37:54 2013
New Revision: 1450835

URL: http://svn.apache.org/r1450835
Log:
MAPREDUCE-5035. Update MR1 memory configuration docs.

Modified:
    hadoop/common/branches/branch-1/CHANGES.txt
    hadoop/common/branches/branch-1/src/docs/src/documentation/content/xdocs/cluster_setup.xml

Modified: hadoop/common/branches/branch-1/CHANGES.txt
URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/CHANGES.txt?rev=1450835&r1=1450834&r2=1450835&view=diff
==============================================================================
--- hadoop/common/branches/branch-1/CHANGES.txt (original)
+++ hadoop/common/branches/branch-1/CHANGES.txt Wed Feb 27 16:37:54 2013
@@ -505,6 +505,8 @@ Release 1.2.0 - unreleased
     MAPREDUCE-5008. Merger progress miscounts with respect to EOF_MARKER.
     (Sandy Ryza via tomwhite)
 
+    MAPREDUCE-5035. Update MR1 memory configuration docs. (tomwhite)
+
 Release 1.1.2 - Unreleased
 
   INCOMPATIBLE CHANGES

Modified: hadoop/common/branches/branch-1/src/docs/src/documentation/content/xdocs/cluster_setup.xml
URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/docs/src/documentation/content/xdocs/cluster_setup.xml?rev=1450835&r1=1450834&r2=1450835&view=diff
==============================================================================
--- hadoop/common/branches/branch-1/src/docs/src/documentation/content/xdocs/cluster_setup.xml
(original)
+++ hadoop/common/branches/branch-1/src/docs/src/documentation/content/xdocs/cluster_setup.xml
Wed Feb 27 16:37:54 2013
@@ -721,7 +721,7 @@
         A TT ensures that a task is killed if it, and 
         its descendants, use VMEM over the task's per-task limit. It also 
         ensures that one or more tasks are killed if the sum total of VMEM 
-        usage by all tasks, and their descendents, cross the node-limit.</p>
+        usage by all tasks, and their descendants, cross the node-limit.</p>
         
         <p>Users can, optionally, specify the VMEM task-limit per job. If no
         such limit is provided, a default limit is used. A node-limit can be 
@@ -733,20 +733,25 @@
 
         <table>
           <tr><th>Name</th><th>Type</th><th>Description</th></tr>
-          <tr><td>mapred.tasktracker.vmem.reserved</td><td>long</td>
-            <td>A number, in bytes, that represents an offset. The total VMEM on 
-            the machine, minus this offset, is the VMEM node-limit for all 
-            tasks, and their descendants, spawned by the TT. 
+          <tr><td><code>mapred.cluster.map.memory.mb</code>, <code>mapred.cluster.reduce.memory.mb</code></td><td>long</td>
+            <td>The size, in terms of virtual memory, of a single map/reduce slot
+                in the Map-Reduce framework, used by the scheduler.
+                A job can ask for multiple slots for a single task via
+                mapred.job.map.memory.mb/mapred.job.reduce.memory.mb, up to the limit specified
by
+                mapred.cluster.max.map.memory.mb/mapred.cluster.max.reduce.memory.mb, if
the scheduler supports the feature.
+                The value of -1 indicates that this feature is turned off.
           </td></tr>
-          <tr><td>mapred.task.default.maxvmem</td><td>long</td>
+          <tr><td><code>mapred.job.map.memory.mb</code>, <code>mapred.job.reduce.memory.mb</code></td><td>long</td>
             <td>A number, in bytes, that represents the default VMEM task-limit 
-            associated with a task. Unless overridden by a job's setting, 
-            this number defines the VMEM task-limit.   
+            associated with a map/reduce task. Unless overridden by a job's setting,
+            this number defines the VMEM task-limit. These properties replace the old deprecated
property,
+                <code>mapred.task.default.maxvmem</code>.
           </td></tr>
-          <tr><td>mapred.task.limit.maxvmem</td><td>long</td>
+          <tr><td><code>mapred.cluster.max.map.memory.mb</code>,
<code>mapred.cluster.max.reduce.memory.mb</code></td><td>long</td>
             <td>A number, in bytes, that represents the upper VMEM task-limit 
-            associated with a task. Users, when specifying a VMEM task-limit 
-            for their tasks, should not specify a limit which exceeds this amount. 
+            associated with a map/reduce task. Users, when specifying a VMEM task-limit
+            for their tasks, should not specify a limit which exceeds this amount. These
properties replace the old deprecated property,
+                <code>mapred.task.limit.maxvmem</code>.
           </td></tr>
         </table>
         
@@ -754,7 +759,7 @@
 
     <table>
           <tr><th>Name</th><th>Type</th><th>Description</th></tr>
-          <tr><td>mapred.tasktracker.taskmemorymanager.monitoring-interval</td>
+          <tr><td><code>mapred.tasktracker.taskmemorymanager.monitoring-interval</code></td>
             <td>long</td>
             <td>The time interval, in milliseconds, between which the TT 
             checks for any memory violation. The default value is 5000 msec
@@ -768,14 +773,6 @@
           above are missing or -1 is specified , memory monitoring is 
           disabled for the TT.
           </li>
-          <li>In addition, monitoring is disabled if 
-          <code>mapred.task.default.maxvmem</code> is greater than 
-          <code>mapred.task.limit.maxvmem</code>. 
-          </li>
-          <li>If a TT receives a task whose task-limit is set by the user
-          to a value larger than <code>mapred.task.limit.maxvmem</code>, it 
-          logs a warning but executes the task.
-          </li> 
           <li>Periodically, the TT checks the following: 
           <ul>
             <li>If any task's current VMEM usage is greater than that task's
@@ -786,7 +783,7 @@
             <li>If the sum total of VMEM used by all tasks and descendants is 
             greater than the node-limit, the TT kills enough tasks, in the
             order of least progress made, till the overall VMEM usage falls
-            below the node-limt. Such killed tasks are not considered failed
+            below the node-limit. Such killed tasks are not considered failed
             and their killing does not count towards the tasks' failure counts.
             </li>
           </ul>
@@ -798,31 +795,15 @@
         tasks only if the TT has enough VMEM free. In addition, Schedulers may 
         choose to consider the physical memory (RAM) available on the node
         as well. To enable Scheduler support, TTs report their memory settings 
-        to the JobTracker in every heartbeat. Before getting into details, 
-        consider the following additional memory-related parameters than can be 
-        configured to enable better scheduling:</p> 
-
-        <table>
-          <tr><th>Name</th><th>Type</th><th>Description</th></tr>
-          <tr><td>mapred.tasktracker.pmem.reserved</td><td>int</td>
-            <td>A number, in bytes, that represents an offset. The total 
-            physical memory (RAM) on the machine, minus this offset, is the 
-            recommended RAM node-limit. The RAM node-limit is a hint to a
-            Scheduler to scheduler only so many tasks such that the sum 
-            total of their RAM requirements does not exceed this limit. 
-            RAM usage is not monitored by a TT.   
-          </td></tr>
-        </table>
+        to the JobTracker in every heartbeat.</p>
         
         <p>A TT reports the following memory-related numbers in every 
         heartbeat:</p>
         <ul>
           <li>The total VMEM available on the node.</li>
-          <li>The value of <code>mapred.tasktracker.vmem.reserved</code>,
-           if set.</li>
+          <li>The remaining VMEM available on the node.</li>
           <li>The total RAM available on the node.</li> 
-          <li>The value of <code>mapred.tasktracker.pmem.reserved</code>,
-           if set.</li>
+          <li>The remaining RAM available on the node.</li>
          </ul>
         </section>
         



Mime
View raw message