hadoop-mapreduce-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From cdoug...@apache.org
Subject svn commit: r823227 - in /hadoop/mapreduce/trunk: CHANGES.txt src/contrib/gridmix/README src/docs/src/documentation/content/xdocs/gridmix.xml src/docs/src/documentation/content/xdocs/site.xml
Date Thu, 08 Oct 2009 16:41:32 GMT
Author: cdouglas
Date: Thu Oct  8 16:41:31 2009
New Revision: 823227

URL: http://svn.apache.org/viewvc?rev=823227&view=rev
Log:
MAPREDUCE-1063. Document gridmix benchmark.

Added:
    hadoop/mapreduce/trunk/src/contrib/gridmix/README
    hadoop/mapreduce/trunk/src/docs/src/documentation/content/xdocs/gridmix.xml
Modified:
    hadoop/mapreduce/trunk/CHANGES.txt
    hadoop/mapreduce/trunk/src/docs/src/documentation/content/xdocs/site.xml

Modified: hadoop/mapreduce/trunk/CHANGES.txt
URL: http://svn.apache.org/viewvc/hadoop/mapreduce/trunk/CHANGES.txt?rev=823227&r1=823226&r2=823227&view=diff
==============================================================================
--- hadoop/mapreduce/trunk/CHANGES.txt (original)
+++ hadoop/mapreduce/trunk/CHANGES.txt Thu Oct  8 16:41:31 2009
@@ -433,6 +433,8 @@
     MAPREDUCE-639. Change Terasort example to reflect the 2009 updates. 
     (omalley)
 
+    MAPREDUCE-1063. Document gridmix benchmark. (cdouglas)
+
   BUG FIXES
 
     MAPREDUCE-878. Rename fair scheduler design doc to 

Added: hadoop/mapreduce/trunk/src/contrib/gridmix/README
URL: http://svn.apache.org/viewvc/hadoop/mapreduce/trunk/src/contrib/gridmix/README?rev=823227&view=auto
==============================================================================
--- hadoop/mapreduce/trunk/src/contrib/gridmix/README (added)
+++ hadoop/mapreduce/trunk/src/contrib/gridmix/README Thu Oct  8 16:41:31 2009
@@ -0,0 +1,22 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+This project implements the third version of Gridmix, a benchmark for live
+clusters. Given a description of jobs (a "trace") annotated with information
+about I/O, memory, etc. a synthetic mix of jobs will be generated and submitted
+to the cluster.
+
+Documentation of usage and configuration properties in forrest is available in
+src/docs/src/documentation/content/xdocs/gridmix.xml

Added: hadoop/mapreduce/trunk/src/docs/src/documentation/content/xdocs/gridmix.xml
URL: http://svn.apache.org/viewvc/hadoop/mapreduce/trunk/src/docs/src/documentation/content/xdocs/gridmix.xml?rev=823227&view=auto
==============================================================================
--- hadoop/mapreduce/trunk/src/docs/src/documentation/content/xdocs/gridmix.xml (added)
+++ hadoop/mapreduce/trunk/src/docs/src/documentation/content/xdocs/gridmix.xml Thu Oct  8
16:41:31 2009
@@ -0,0 +1,164 @@
+<?xml version="1.0"?>
+<!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+
+<!DOCTYPE document PUBLIC "-//APACHE//DTD Documentation V2.0//EN" "http://forrest.apache.org/dtd/document-v20.dtd">
+
+<document>
+
+<header>
+  <title>Gridmix</title>
+</header>
+
+<body>
+
+  <section>
+  <title>Overview</title>
+
+  <p>Gridmix is a benchmark for live clusters. It submits a mix of synthetic
+  jobs, modeling a profile mined from production loads.</p>
+
+  <p>There exist three versions of the Gridmix tool. This document discusses
+  the third (checked into contrib), distinct from the two checked into the
+  benchmarks subdirectory. While the first two versions of the tool included
+  stripped-down versions of common jobs, both were principally saturation
+  tools for stressing the framework at scale. In support of a broader range of
+  deployments and finer-tuned job mixes, this version of the tool will attempt
+  to model the resource profiles of production jobs to identify bottlenecks,
+  guide development, and serve as a replacement for the existing gridmix
+  benchmarks.</p>
+
+  </section>
+
+  <section id="usage">
+
+  <title>Usage</title>
+
+  <p>To run Gridmix, one requires a job trace describing the job mix for a
+  given cluster. Such traces are typically genenerated by Rumen (see related
+  documentation). Gridmix also requires input data from which the synthetic
+  jobs will draw bytes. The input data need not be in any particular format,
+  as the synthetic jobs are currently binary readers. If one is running on a
+  new cluster, an optional step generating input data may precede the run.</p>
+
+  <p>Basic command line usage:</p>
+<source>
+
+bin/mapred org.apache.hadoop.mapred.gridmix.Gridmix [-generate &lt;MiB&gt;] &lt;iopath&gt;
&lt;trace&gt;
+</source>
+
+  <p>The <code>-generate</code> parameter accepts standard units, e.g.
+  <code>100g</code> will generate 100 * 2<sup>30</sup> bytes. The
+  &lt;iopath&gt; parameter is the destination directory for generated and/or
+  the directory from which input data will be read. The &lt;trace&gt;
+  parameter is a path to a job trace. The following configuration parameters
+  are also accepted in the standard idiom, before other Gridmix
+  parameters.</p>
+
+  <section>
+  <title>Configuration parameters</title>
+  <p></p>
+  <table>
+    <tr><th> Parameter </th><th> Description </th><th>
Notes </th></tr>
+    <tr><td><code>gridmix.output.directory</code></td>
+        <td>The directory into which output will be written. If specified, the
+        <code>iopath</code> will be relative to this parameter.</td>
+        <td>The submitting user must have read/write access to this
+        directory. The user should also be mindful of any quota issues that
+        may arise during a run.</td></tr>
+    <tr><td><code>gridmix.client.submit.threads</code></td>
+        <td>The number of threads submitting jobs to the cluster. This also
+        controls how many splits will be loaded into memory at a given time,
+        pending the submit time in the trace.</td>
+        <td>Splits are pregenerated to hit submission deadlines, so
+        particularly dense traces may want more submitting threads. However,
+        storing splits in memory is reasonably expensive, so one should raise
+        this cautiously.</td></tr>
+    <tr><td><code>gridmix.client.pending.queue.depth</code></td>
+        <td>The depth of the queue of job descriptions awaiting split
+        generation.</td>
+        <td>The jobs read from the trace occupy a queue of this depth before
+        being processed by the submission threads. It is unusual to configure
+        this.</td></tr>
+    <tr><td><code>gridmix.min.key.length</code></td>
+        <td>The key size for jobs submitted to the cluster.</td>
+        <td>While this is clearly a job-specific, even task-specific property,
+        no data on key length is currently available. Since the intermediate
+        data are random, memcomparable data, not even the sort is likely
+        affected. It exists as a tunable as no default value is appropriate,
+        but future versions will likely replace it with trace data.</td></tr>
+  </table>
+
+  </section>
+</section>
+
+<section id="assumptions">
+
+  <title>Simplifying Assumptions</title>
+
+  <p>Gridmix will be developed in stages, incorporating feedback and patches
+  from the community. Currently, its intent is to evaluate Map/Reduce and HDFS
+  performance and not the layers on top of them (i.e. the extensive lib and
+  subproject space). Given these two limitations, the following
+  characteristics of job load are not currently captured in job traces and
+  cannot be accurately reproduced in Gridmix.</p>
+
+  <table>
+  <tr><th>Property</th><th>Notes</th></tr>
+  <tr><td>CPU usage</td><td>We have no data for per-task CPU usage,
so we
+  cannot attempt even an approximation. Gridmix tasks are never CPU bound
+  independent of I/O, though this surely happens in practice.</td></tr>
+  <tr><td>Filesystem properties</td><td>No attempt is made to match
block
+  sizes, namespace hierarchies, or any property of input, intermediate, or
+  output data other than the bytes/records consumed and emitted from a given
+  task. This implies that some of the most heavily used parts of the system-
+  the compression libraries, text processing, streaming, etc.- cannot be
+  meaningfully tested with the current implementation.</td></tr>
+  <tr><td>I/O rates</td><td>The rate at which records are consumed/emitted
is
+  assumed to be limited only by the speed of the reader/writer and constant
+  throughout the task.</td></tr>
+  <tr><td>Memory profile</td><td>No data on tasks' memory usage over
time is
+  available, though the max heap size is retained.</td></tr>
+  <tr><td>Skew</td><td>The records consumed and emitted to/from a
given task
+  are assumed to follow observed averages, i.e. records will be more regular
+  than may be seen in the wild. Each map also generates a proportional
+  percentage of data for each reduce, so a job with unbalanced input will be
+  flattened.</td></tr>
+  <tr><td>Job failure</td><td>User code is assumed to be correct.</td></tr>
+  <tr><td>Job independence</td><td>The output or outcome of one job
does not
+  affect when or whether a subsequent job will run.</td></tr>
+  </table>
+
+</section>
+
+<section>
+
+  <title>Appendix</title>
+
+  <p>Issues tracking the implementations of <a
+  href="https://issues.apache.org/jira/browse/HADOOP-2369">gridmix1</a>, <a
+  href="https://issues.apache.org/jira/browse/HADOOP-3770">gridmix2</a>, and
+  <a href="https://issues.apache.org/jira/browse/MAPREDUCE-776">gridmix3</a>.
+  Other issues tracking the development of Gridmix can be found by searching
+  the Map/Reduce <a
+  href="https://issues.apache.org/jira/browse/MAPREDUCE">JIRA</a></p>
+
+</section>
+
+</body>
+
+</document>

Modified: hadoop/mapreduce/trunk/src/docs/src/documentation/content/xdocs/site.xml
URL: http://svn.apache.org/viewvc/hadoop/mapreduce/trunk/src/docs/src/documentation/content/xdocs/site.xml?rev=823227&r1=823226&r2=823227&view=diff
==============================================================================
--- hadoop/mapreduce/trunk/src/docs/src/documentation/content/xdocs/site.xml (original)
+++ hadoop/mapreduce/trunk/src/docs/src/documentation/content/xdocs/site.xml Thu Oct  8 16:41:31
2009
@@ -43,6 +43,7 @@
 		<distcp    					label="DistCp"       href="distcp.html" />
 		<vaidya    					label="Vaidya" 		href="vaidya.html"/>
 		<archives  				label="Hadoop Archives"     href="hadoop_archives.html"/>
+		<gridmix  				label="Gridmix"     href="gridmix.html"/>
    </docs>
    
     <docs label="Schedulers">



Mime
View raw message