hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From a.@apache.org
Subject hadoop git commit: HADOOP-13107. clean up how rumen is executed [Forced Update!]
Date Fri, 06 May 2016 15:58:12 GMT
Repository: hadoop
Updated Branches:
  refs/heads/HADOOP-12930 0d5505ad7 -> 9a1f4983d (forced update)


HADOOP-13107. clean up how rumen is executed


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/9a1f4983
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/9a1f4983
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/9a1f4983

Branch: refs/heads/HADOOP-12930
Commit: 9a1f4983d4dedd68d8f8ec154833cc43ff6f4e12
Parents: e3d89a1
Author: Allen Wittenauer <aw@apache.org>
Authored: Fri May 6 08:47:31 2016 -0700
Committer: Allen Wittenauer <aw@apache.org>
Committed: Fri May 6 08:57:53 2016 -0700

----------------------------------------------------------------------
 .../main/resources/assemblies/hadoop-tools.xml  |  8 +++
 .../src/main/shellprofile.d/hadoop-rumen.sh     | 58 ++++++++++++++++++++
 .../hadoop-rumen/src/site/markdown/Rumen.md.vm  | 40 +++-----------
 3 files changed, 75 insertions(+), 31 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/hadoop/blob/9a1f4983/hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml
----------------------------------------------------------------------
diff --git a/hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml b/hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml
index c5ea6ad..8606e23 100644
--- a/hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml
+++ b/hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml
@@ -133,6 +133,14 @@
       </includes>
     </fileSet>
     <fileSet>
+      <directory>../hadoop-rumen/src/main/shellprofile.d</directory>
+      <includes>
+        <include>*</include>
+      </includes>
+      <outputDirectory>/libexec/shellprofile.d</outputDirectory>
+      <fileMode>0755</fileMode>
+    </fileSet>
+    <fileSet>
       <directory>../hadoop-streaming/target</directory>
       <outputDirectory>/share/hadoop/${hadoop.component}/sources</outputDirectory>
       <includes>

http://git-wip-us.apache.org/repos/asf/hadoop/blob/9a1f4983/hadoop-tools/hadoop-rumen/src/main/shellprofile.d/hadoop-rumen.sh
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-rumen/src/main/shellprofile.d/hadoop-rumen.sh b/hadoop-tools/hadoop-rumen/src/main/shellprofile.d/hadoop-rumen.sh
new file mode 100755
index 0000000..d7d4022
--- /dev/null
+++ b/hadoop-tools/hadoop-rumen/src/main/shellprofile.d/hadoop-rumen.sh
@@ -0,0 +1,58 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+if ! declare -f hadoop_subcommand_rumenfolder >/dev/null 2>/dev/null; then
+
+  if [[ "${HADOOP_SHELL_EXECNAME}" = hadoop ]]; then
+    hadoop_add_subcommand "rumenfolder" "scale a rumen input trace"
+  fi
+
+## @description  rumenfolder command for hadoop
+## @audience     public
+## @stability    stable
+## @replaceable  yes
+function hadoop_subcommand_rumenfolder
+{
+  # shellcheck disable=SC2034
+  HADOOP_CLASSNAME=org.apache.hadoop.tools.rumen.Folder
+  hadoop_add_to_classpath_tools hadoop-rumen
+  hadoop_debug "Appending HADOOP_CLIENT_OPTS onto HADOOP_OPTS"
+  HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_CLIENT_OPTS}"
+}
+
+fi
+
+if ! declare -f hadoop_subcommand_rumentrace >/dev/null 2>/dev/null; then
+
+  if [[ "${HADOOP_SHELL_EXECNAME}" = hadoop ]]; then
+    hadoop_add_subcommand "rumentrace" "convert logs into a rumen trace"
+  fi
+
+## @description  rumentrace command for hadoop
+## @audience     public
+## @stability    stable
+## @replaceable  yes
+function hadoop_subcommand_rumentrace
+{
+  # shellcheck disable=SC2034
+  HADOOP_CLASSNAME=org.apache.hadoop.tools.rumen.TraceBuilder
+  hadoop_add_to_classpath_tools hadoop-rumen
+  hadoop_debug "Appending HADOOP_CLIENT_OPTS onto HADOOP_OPTS"
+  HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_CLIENT_OPTS}"
+}
+
+fi

http://git-wip-us.apache.org/repos/asf/hadoop/blob/9a1f4983/hadoop-tools/hadoop-rumen/src/site/markdown/Rumen.md.vm
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-rumen/src/site/markdown/Rumen.md.vm b/hadoop-tools/hadoop-rumen/src/site/markdown/Rumen.md.vm
index bee976a..34dfd0b 100644
--- a/hadoop-tools/hadoop-rumen/src/site/markdown/Rumen.md.vm
+++ b/hadoop-tools/hadoop-rumen/src/site/markdown/Rumen.md.vm
@@ -50,8 +50,8 @@ but a simulation of the scheduler elects to run that task on a remote
 rack, the simulator requires a runtime its input cannot provide. 
 To fill in these gaps, Rumen performs a statistical analysis of the 
 digest to estimate the variables the trace doesn't supply. Rumen traces 
-drive both Gridmix (a benchmark of Hadoop MapReduce clusters) and Mumak 
-(a simulator for the JobTracker).
+drive both Gridmix (a benchmark of Hadoop MapReduce clusters) and SLS
+(a simulator for the resource manager scheduler).
 
 
 $H3 Motivation
@@ -126,16 +126,13 @@ can use the `Folder` utility to fold the current trace to the
 desired length. The remaining part of this section explains these 
 utilities in detail.
     
-Examples in this section assumes that certain libraries are present 
-in the java CLASSPATH. See [Dependencies](#Dependencies) for more details.
-
 
 $H3 Trace Builder
       
 $H4 Command
 
 ```
-java org.apache.hadoop.tools.rumen.TraceBuilder [options] <jobtrace-output> <topology-output>
<inputs>
+hadoop rumentrace [options] <jobtrace-output> <topology-output> <inputs>
 ```
   
 This command invokes the `TraceBuilder` utility of *Rumen*.
@@ -205,12 +202,8 @@ $H4 Options
 
 $H4 Example
 
-*Rumen* expects certain library *JARs* to be present in  the *CLASSPATH*.
-One simple way to run Rumen is to use
-`$HADOOP_HOME/bin/hadoop jar` command to run it as example below.
-
 ```
-java org.apache.hadoop.tools.rumen.TraceBuilder \
+hadoop rumentrace \
   file:///tmp/job-trace.json \
   file:///tmp/job-topology.json \
   hdfs:///tmp/hadoop-yarn/staging/history/done_intermediate/testuser
@@ -229,7 +222,7 @@ $H3 Folder
 $H4 Command
 
 ```
-java org.apache.hadoop.tools.rumen.Folder [options] [input] [output]
+hadoop rumenfolder [options] [input] [output]
 ```
       
 This command invokes the `Folder` utility of 
@@ -350,7 +343,7 @@ $H4 Examples
 $H5 Folding an input trace with 10 hours of total runtime to generate an output trace with
1 hour of total runtime
 
 ```
-java org.apache.hadoop.tools.rumen.Folder \
+hadoop rumenfolder \
   -output-duration 1h \
   -input-cycle 20m \
   file:///tmp/job-trace.json \
@@ -362,7 +355,7 @@ If the folded jobs are out of order then the command will bail out.
 $H5 Folding an input trace with 10 hours of total runtime to generate an output trace with
1 hour of total runtime and tolerate some skewness
 
 ```
-java org.apache.hadoop.tools.rumen.Folder \
+hadoop rumenfolder \
   -output-duration 1h \
   -input-cycle 20m \
   -allow-missorting \
@@ -378,7 +371,7 @@ If the folded jobs are out of order, then atmost
 $H5 Folding an input trace with 10 hours of total runtime to generate an output trace with
1 hour of total runtime in debug mode
 
 ```
-java org.apache.hadoop.tools.rumen.Folder \
+hadoop rumenfolder \
   -output-duration 1h \
   -input-cycle 20m \
   -debug -temp-directory file:///tmp/debug \
@@ -395,7 +388,7 @@ up.
 $H5 Folding an input trace with 10 hours of total runtime to generate an output trace with
1 hour of total runtime with custom concentration.
 
 ```
-java org.apache.hadoop.tools.rumen.Folder \
+hadoop rumenfolder \
   -output-duration 1h \
   -input-cycle 20m \
   -concentration 2 \
@@ -421,18 +414,3 @@ Look at the MapReduce
 <a href="https://issues.apache.org/jira/browse/MAPREDUCE/component/12313617">rumen-component</a>
 for further details.
 
-
-$H3 Dependencies
-
-*Rumen* expects certain library *JARs* to be present in  the *CLASSPATH*.
-One simple way to run Rumen is to use
-`hadoop jar` command to run it as example below.
-
-```
-$HADOOP_HOME/bin/hadoop jar \
-  $HADOOP_HOME/share/hadoop/tools/lib/hadoop-rumen-2.5.1.jar \
-  org.apache.hadoop.tools.rumen.TraceBuilder \
-  file:///tmp/job-trace.json \
-  file:///tmp/job-topology.json \
-  hdfs:///tmp/hadoop-yarn/staging/history/done_intermediate/testuser
-```


---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org


Mime
View raw message