hadoop-mapreduce-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vino...@apache.org
Subject svn commit: r1143803 - in /hadoop/common/branches/MR-279/mapreduce: CHANGES.txt INSTALL yarn/README
Date Thu, 07 Jul 2011 12:54:42 GMT
Author: vinodkv
Date: Thu Jul  7 12:54:41 2011
New Revision: 1143803

URL: http://svn.apache.org/viewvc?rev=1143803&view=rev
Log:
MAPREDUCE-2645. Updates to MRv2 INSTALL and README documentation. Contributed by Josh Wills.

Modified:
    hadoop/common/branches/MR-279/mapreduce/CHANGES.txt
    hadoop/common/branches/MR-279/mapreduce/INSTALL
    hadoop/common/branches/MR-279/mapreduce/yarn/README

Modified: hadoop/common/branches/MR-279/mapreduce/CHANGES.txt
URL: http://svn.apache.org/viewvc/hadoop/common/branches/MR-279/mapreduce/CHANGES.txt?rev=1143803&r1=1143802&r2=1143803&view=diff
==============================================================================
--- hadoop/common/branches/MR-279/mapreduce/CHANGES.txt (original)
+++ hadoop/common/branches/MR-279/mapreduce/CHANGES.txt Thu Jul  7 12:54:41 2011
@@ -5,6 +5,9 @@ Trunk (unreleased changes)
 
     MAPREDUCE-279
 
+    MAPREDUCE-2645. Updates to MRv2 INSTALL and README documentation.
+    (Josh Wills via vinodkv)
+
     MAPREDUCE-2633. Add a getCounter(Enum) method to the Counters record. 
     (Josh Wills via sharad)
 

Modified: hadoop/common/branches/MR-279/mapreduce/INSTALL
URL: http://svn.apache.org/viewvc/hadoop/common/branches/MR-279/mapreduce/INSTALL?rev=1143803&r1=1143802&r2=1143803&view=diff
==============================================================================
--- hadoop/common/branches/MR-279/mapreduce/INSTALL (original)
+++ hadoop/common/branches/MR-279/mapreduce/INSTALL Thu Jul  7 12:54:41 2011
@@ -29,7 +29,7 @@ ant veryclean jar jar-test  -Dresolvers=
 
 In case you want to skip the tests run:
 
-mvn clean install assembly:assembly -Dmaven.test.skip.exec=true
+mvn clean install assembly:assembly -DskipTests
 ant veryclean jar jar-test  -Dresolvers=internal 
 
 You will see a tarball in
@@ -55,7 +55,9 @@ Step 9) bin/yarn-daemon.sh start resourc
 
 Step 10) bin/yarn-daemon.sh start nodemanager
 
-Step 11) Create the following symlinks in hadoop-common/lib 
+Step 11) bin/yarn-daemon.sh start historyserver
+
+Step 12) Create the following symlinks in hadoop-common/lib 
 
 ln -s $HADOOP_YARN_INSTALL/modules/hadoop-mapreduce-client-app-1.0-SNAPSHOT.jar .	
 ln -s $HADOOP_YARN_INSTALL/modules/yarn-api-1.0-SNAPSHOT.jar .
@@ -66,8 +68,22 @@ ln -s $HADOOP_YARN_INSTALL/modules/yarn-
 ln -s $HADOOP_YARN_INSTALL/modules/hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT.jar .
 ln -s $HADOOP_YARN_INSTALL/lib/protobuf-java-2.4.0a.jar .
 
-Step 12) You are all set, an example on how to run a job is:
-
-$HADOOP_COMMON_HOME/bin/hadoop jar $HADOOP_MAPRED_HOME/build/hadoop-mapred-examples-0.22.0-SNAPSHOT.jar
randomwriter -Dmapreduce.job.user.name=$USER -Dmapreduce.randomwriter.bytespermap=10000 -Ddfs.blocksize=536870912
-Ddfs.block.size=536870912 -libjars $HADOOP_YARN_INSTALL/hadoop-mapreduce-1.0-SNAPSHOT/modules/hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT.jar
output 
+Step 13) Yarn daemons are up! But for running mapreduce applications, which now are in user
land, you need to setup nodemanager with the following configuration in your yarn-site.xml
before you start the nodemanager.
+    <property>
+      <name>nodemanager.auxiluary.services</name>
+      <value>mapreduce.shuffle</value>
+    </property>
+
+    <property>
+      <name>nodemanager.aux.service.mapreduce.shuffle.class</name>
+      <value>org.apache.hadoop.mapred.ShuffleHandler</value>
+    </property>
+
+Step 14) You are all set, an example on how to run a mapreduce job is:
+
+cd $HADOOP_MAPRED_HOME
+ant examples -Dresolvers=internal
+$HADOOP_COMMON_HOME/bin/hadoop jar $HADOOP_MAPRED_HOME/build/hadoop-mapred-examples-0.22.0-SNAPSHOT.jar
randomwriter -Dmapreduce.job.user.name=$USER -Dmapreduce.clientfactory.class.name=org.apache.hadoop.mapred.YarnClientFactory
-Dmapreduce.randomwriter.bytespermap=10000 -Ddfs.blocksize=536870912 -Ddfs.block.size=536870912
-libjars $HADOOP_YARN_INSTALL/hadoop-mapreduce-1.0-SNAPSHOT/modules/hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT.jar
output 
 
+The output on the command line should be almost similar to what you see in the JT/TT setup
(Hadoop 0.20/0.21)
 

Modified: hadoop/common/branches/MR-279/mapreduce/yarn/README
URL: http://svn.apache.org/viewvc/hadoop/common/branches/MR-279/mapreduce/yarn/README?rev=1143803&r1=1143802&r2=1143803&view=diff
==============================================================================
--- hadoop/common/branches/MR-279/mapreduce/yarn/README (original)
+++ hadoop/common/branches/MR-279/mapreduce/yarn/README Thu Jul  7 12:54:41 2011
@@ -8,9 +8,6 @@ Maven: Maven 2
 
 Setup
 -----
-Install the Avro maven plugin in local maven repository:
-mvn install:install-file -Dfile=./avro-maven-plugin/avro-maven-plugin-1.4.0-SNAPSHOT.jar
-DpomFile=./avro-maven-plugin/avro-maven-plugin-1.4.0-SNAPSHOT.pom
-
 Install protobuf 2.4.0a or higher (Download from http://code.google.com/p/protobuf/downloads/list)
  - install the protoc executable (configure, make, make install)
  - install the maven artifact (cd java; mvn install)
@@ -72,32 +69,5 @@ yarn-common - Utilities which can be use
 
 yarn-server - Implementation of the yarn-api
 	yarn-server-common - APIs shared between resourcemanager and nodemanager
-	yarn-server-nodemanager
-	yarn-server-resourcemanager
-
-yarn-prototype - AppMaster (Prototype) to launch TTs via nodemanager
-
-yarn-mapreduce - Mapreduce specific parts
-	yarn-mapreduce-common - Mapreduce utilities and interfae shared between client and mapreduce
AppMaster
-	yarn-mapreduce-client - Mapreduce client (uses JobClient)
-	yarn-mapreduce-app - Mapreduce AppMaster
-
-yarn-master-worker - master-worker AppMaster
-
-General instructions for starting Yarn and MapReduce
-----------------------------------------------------
-At yarn project root,
-
-	mvn clean install
-
-ResourceManager:
-	cd yarn-server/yarn-server-resourcemanager
-	mvn exec:java -Dexec.mainClass=org.apache.hadoop.yarn.server.resourcemanager.ResourceManager
-
-NodeManager:
-	cd yarn-server/yarn-server-nodemanager
-	mvn exec:java -Dexec.mainClass=org.apache.hadoop.yarn.server.nodemanager.NodeManager
-
-MRAppMaster
-	cd yarn-mapreduce/yarn-mapreduce-app
-	mvn exec:java -Dexec.mainClass=org.apache.hadoop.yarn.mapreduce.app.MRAppMaster -Dexec.args="<clusterts>
<appid>"
+	yarn-server-nodemanager (TaskTracker replacement)
+	yarn-server-resourcemanager (JobTracker replacement)



Mime
View raw message