hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From z..@apache.org
Subject [7/9] hadoop git commit: MAPREDUCE-6151. Update document of GridMix (Masatake Iwasaki via aw)
Date Fri, 30 Jan 2015 21:43:51 GMT
MAPREDUCE-6151. Update document of GridMix (Masatake Iwasaki via aw)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/fa381070
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/fa381070
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/fa381070

Branch: refs/heads/HDFS-EC
Commit: fa38107026041f5dcd864240277399fb1d86a987
Parents: cec3166
Author: Allen Wittenauer <aw@apache.org>
Authored: Fri Jan 30 11:08:20 2015 -0800
Committer: Zhe Zhang <zhz@apache.org>
Committed: Fri Jan 30 13:42:05 2015 -0800

----------------------------------------------------------------------
 hadoop-mapreduce-project/CHANGES.txt            |   2 +
 hadoop-project/src/site/site.xml                |   1 +
 .../src/site/markdown/GridMix.md.vm             | 107 ++++++++++---------
 3 files changed, 57 insertions(+), 53 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/hadoop/blob/fa381070/hadoop-mapreduce-project/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/CHANGES.txt b/hadoop-mapreduce-project/CHANGES.txt
index 496913f..7672f08 100644
--- a/hadoop-mapreduce-project/CHANGES.txt
+++ b/hadoop-mapreduce-project/CHANGES.txt
@@ -266,6 +266,8 @@ Release 2.7.0 - UNRELEASED
 
     MAPREDUCE-6150. Update document of Rumen (Masatake Iwasaki via aw)
 
+    MAPREDUCE-6151. Update document of GridMix (Masatake Iwasaki via aw)
+
   OPTIMIZATIONS
 
     MAPREDUCE-6169. MergeQueue should release reference to the current item 

http://git-wip-us.apache.org/repos/asf/hadoop/blob/fa381070/hadoop-project/src/site/site.xml
----------------------------------------------------------------------
diff --git a/hadoop-project/src/site/site.xml b/hadoop-project/src/site/site.xml
index 113cb13..79136e2 100644
--- a/hadoop-project/src/site/site.xml
+++ b/hadoop-project/src/site/site.xml
@@ -105,6 +105,7 @@
       <item name="Hadoop Streaming" href="hadoop-mapreduce-client/hadoop-mapreduce-client-core/HadoopStreaming.html"/>
       <item name="Hadoop Archives" href="hadoop-mapreduce-client/hadoop-mapreduce-client-core/HadoopArchives.html"/>
       <item name="DistCp" href="hadoop-mapreduce-client/hadoop-mapreduce-client-core/DistCp.html"/>
+      <item name="GridMix" href="hadoop-gridmix/GridMix.html"/>
       <item name="Rumen" href="hadoop-rumen/Rumen.html"/>
     </menu>
 

http://git-wip-us.apache.org/repos/asf/hadoop/blob/fa381070/hadoop-tools/hadoop-gridmix/src/site/markdown/GridMix.md.vm
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-gridmix/src/site/markdown/GridMix.md.vm b/hadoop-tools/hadoop-gridmix/src/site/markdown/GridMix.md.vm
index 53c8891..5e4199b 100644
--- a/hadoop-tools/hadoop-gridmix/src/site/markdown/GridMix.md.vm
+++ b/hadoop-tools/hadoop-gridmix/src/site/markdown/GridMix.md.vm
@@ -38,21 +38,14 @@ Overview
 
 GridMix is a benchmark for Hadoop clusters. It submits a mix of
 synthetic jobs, modeling a profile mined from production loads.
-
-There exist three versions of the GridMix tool. This document
-discusses the third (checked into `src/contrib` ), distinct
-from the two checked into the `src/benchmarks` sub-directory.
-While the first two versions of the tool included stripped-down versions
-of common jobs, both were principally saturation tools for stressing the
-framework at scale. In support of a broader range of deployments and
-finer-tuned job mixes, this version of the tool will attempt to model
+This version of the tool will attempt to model
 the resource profiles of production jobs to identify bottlenecks, guide
-development, and serve as a replacement for the existing GridMix
-benchmarks.
+development.
 
 To run GridMix, you need a MapReduce job trace describing the job mix
-for a given cluster. Such traces are typically generated by Rumen (see
-Rumen documentation). GridMix also requires input data from which the
+for a given cluster. Such traces are typically generated by
+[Rumen](../hadoop-rumen/Rumen.html).
+GridMix also requires input data from which the
 synthetic jobs will be reading bytes. The input data need not be in any
 particular format, as the synthetic jobs are currently binary readers.
 If you are running on a new cluster, an optional step generating input
@@ -62,10 +55,15 @@ on the same or another cluster, follow these steps:
 
 1.  Locate the job history files on the production cluster. This
     location is specified by the
-    `mapred.job.tracker.history.completed.location`
+    `mapreduce.jobhistory.done-dir` or 
+    `mapreduce.jobhistory.intermediate-done-dir`
     configuration property of the cluster.
-
-2.  Run Rumen to build a job trace in JSON format for all or select jobs.
+    ([MapReduce historyserver](../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredCommands.html#historyserver)
+    moves job history files from `mapreduce.jobhistory.done-dir`
+    to `mapreduce.jobhistory.intermediate-done-dir`.)
+    
+2.  Run [Rumen](../hadoop-rumen/Rumen.html)
+    to build a job trace in JSON format for all or select jobs.
 
 3.  Use GridMix with the job trace on the benchmark cluster.
 
@@ -79,13 +77,17 @@ Usage
 
 Basic command-line usage without configuration parameters:
 
-    org.apache.hadoop.mapred.gridmix.Gridmix [-generate <size>] [-users <users-list>]
<iopath> <trace>
+```
+java org.apache.hadoop.mapred.gridmix.Gridmix [-generate <size>] [-users <users-list>]
<iopath> <trace>
+```
 
 Basic command-line usage with configuration parameters:
 
-    org.apache.hadoop.mapred.gridmix.Gridmix \
-      -Dgridmix.client.submit.threads=10 -Dgridmix.output.directory=foo \
-      [-generate <size>] [-users <users-list>] <iopath> <trace>
+```
+java org.apache.hadoop.mapred.gridmix.Gridmix \
+  -Dgridmix.client.submit.threads=10 -Dgridmix.output.directory=foo \
+  [-generate <size>] [-users <users-list>] <iopath> <trace>
+```
 
 > Configuration parameters like
 > `-Dgridmix.client.submit.threads=10` and
@@ -102,6 +104,8 @@ The `-generate` option is used to generate input data and
 Distributed Cache files for the synthetic jobs. It accepts standard units
 of size suffixes, e.g. `100g` will generate
 100 * 2<sup>30</sup> bytes as input data.
+The minimum size of input data in compressed format (128MB by default)
+is defined by `gridmix.min.file.size`.
 `<iopath>/input` is the destination directory for
 generated input data and/or the directory from which input data will be
 read. HDFS-based Distributed Cache files are generated under the
@@ -121,16 +125,17 @@ uncompressed. Use "-" as the value of this parameter if you
 want to pass an  *uncompressed*  trace via the standard
 input-stream of GridMix.
 
-The class `org.apache.hadoop.mapred.gridmix.Gridmix` can
-be found in the JAR
-`contrib/gridmix/hadoop-gridmix-$VERSION.jar` inside your
-Hadoop installation, where `$VERSION` corresponds to the
-version of Hadoop installed. A simple way of ensuring that this class
-and all its dependencies are loaded correctly is to use the
-`hadoop` wrapper script in Hadoop:
+GridMix expects certain library *JARs* to be present in  the *CLASSPATH*.
+One simple way to run GridMix is to use `hadoop jar` command to run it.
+You also need to add the JAR of Rumen to classpath for both of client and tasks
+as example shown below.
 
-    hadoop jar <gridmix-jar> org.apache.hadoop.mapred.gridmix.Gridmix \
-      [-generate <size>] [-users <users-list>] <iopath> <trace>
+```
+HADOOP_CLASSPATH=$HADOOP_HOME/share/hadoop/tools/lib/hadoop-rumen-2.5.1.jar \
+  $HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/share/hadoop/tools/lib/hadoop-gridmix-2.5.1.jar
\
+    -libjars $HADOOP_HOME/share/hadoop/tools/lib/hadoop-rumen-2.5.1.jar \
+    [-generate <size>] [-users <users-list>] <iopath> <trace>
+```
 
 The supported configuration parameters are explained in the
 following sections.
@@ -262,14 +267,14 @@ recorded in the trace. It constructs jobs of two types:
     </td>
     <td>A synthetic job where each task does  *nothing*  but sleep
     for a certain duration as observed in the production trace. The
-    scalability of the Job Tracker is often limited by how many
+    scalability of the ResourceManager is often limited by how many
     heartbeats it can handle every second. (Heartbeats are periodic
-    messages sent from Task Trackers to update their status and grab new
-    tasks from the Job Tracker.) Since a benchmark cluster is typically
+    messages sent from NodeManagers to update their status and grab new
+    tasks from the ResourceManager.) Since a benchmark cluster is typically
     a fraction in size of a production cluster, the heartbeat traffic
     generated by the slave nodes is well below the level of the
-    production cluster. One possible solution is to run multiple Task
-    Trackers on each slave node. This leads to the obvious problem that
+    production cluster. One possible solution is to run multiple
+    NodeManagers on each slave node. This leads to the obvious problem that
     the I/O workload generated by the synthetic jobs would thrash the
     slave nodes. Hence the need for such a job.</td>
   </tr>
@@ -334,7 +339,7 @@ Job Submission Policies
 
 GridMix controls the rate of job submission. This control can be
 based on the trace information or can be based on statistics it gathers
-from the Job Tracker. Based on the submission policies users define,
+from the ResourceManager. Based on the submission policies users define,
 GridMix uses the respective algorithm to control the job submission.
 There are currently three types of policies:
 
@@ -407,9 +412,9 @@ The following configuration parameters affect the job submission policy:
     <td>
       <code>gridmix.throttle.jobs-to-tracker-ratio</code>
     </td>
-    <td>In STRESS mode, the minimum ratio of running jobs to Task
-    Trackers in a cluster for the cluster to be considered
-     *overloaded* . This is the threshold TJ referred to earlier.
+    <td>In STRESS mode, the minimum ratio of running jobs to
+    NodeManagers in a cluster for the cluster to be considered
+    *overloaded* . This is the threshold TJ referred to earlier.
     The default is 1.0.</td>
   </tr>
   <tr>
@@ -688,20 +693,16 @@ correctly emulate compression.
 Emulating High-Ram jobs
 -----------------------
 
-MapReduce allows users to define a job as a High-Ram job. Tasks from a 
-High-Ram job can occupy multiple slots on the task-trackers. 
-Task-tracker assigns fixed virtual memory for each slot. Tasks from 
-High-Ram jobs can occupy multiple slots and thus can use up more 
-virtual memory as compared to a default task.
-
-Emulating this behavior is important because of the following reasons
+MapReduce allows users to define a job as a High-Ram job. Tasks from a
+High-Ram job can occupy larger fraction of memory in task processes.
+Emulating this behavior is important because of the following reasons.
 
 * Impact on scheduler:  Scheduling of tasks from High-Ram jobs 
-  impacts the scheduling behavior as it might result into slot 
-  reservation and slot/resource utilization.
+  impacts the scheduling behavior as it might result into
+  resource reservation and utilization.
 
-* Impact on the node : Since High-Ram tasks occupy multiple slots,
-  trackers do some bookkeeping for allocating extra resources for 
+* Impact on the node : Since High-Ram tasks occupy larger memory,
+  NodeManagers do some bookkeeping for allocating extra resources for 
   these tasks. Thus this becomes a precursor for memory emulation
   where tasks with high memory requirements needs to be considered
   as a High-Ram task.
@@ -808,11 +809,11 @@ job traces and cannot be accurately reproduced in GridMix:
 Appendix
 --------
 
+There exist older versions of the GridMix tool.
 Issues tracking the original implementations of
-<a href="https://issues.apache.org/jira/browse/HADOOP-2369">GridMix1</a>,
-<a href="https://issues.apache.org/jira/browse/HADOOP-3770">GridMix2</a>,
-and <a href="https://issues.apache.org/jira/browse/MAPREDUCE-776">GridMix3</a>
+[GridMix1](https://issues.apache.org/jira/browse/HADOOP-2369),
+[GridMix2](https://issues.apache.org/jira/browse/HADOOP-3770),
+and [GridMix3](https://issues.apache.org/jira/browse/MAPREDUCE-776)
 can be found on the Apache Hadoop MapReduce JIRA. Other issues tracking
 the current development of GridMix can be found by searching
-<a href="https://issues.apache.org/jira/browse/MAPREDUCE/component/12313086">
-the Apache Hadoop MapReduce JIRA</a>
+[the Apache Hadoop MapReduce JIRA](https://issues.apache.org/jira/browse/MAPREDUCE/component/12313086).


Mime
View raw message