hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Update of "HowToSetupYourDevelopmentEnvironment" by TomWhite
Date Thu, 13 Oct 2011 20:33:24 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The "HowToSetupYourDevelopmentEnvironment" page has been changed by TomWhite:
http://wiki.apache.org/hadoop/HowToSetupYourDevelopmentEnvironment?action=diff&rev1=29&rev2=30

  
  Note that the {{{start-dfs.sh}}} script will not work with this set up, since it assumes
that HADOOP_COMMON_HOME and HADOOP_HDFS_HOME are the same directory.
  
+ = Run MapReduce in pseudo-distributed mode from the dev tree =
+ 
+ Build the packaging from the top level. This will build the distribution in an exploded
format that we can run directly (i.e. no need to untar):
+ {{{
+ mvn clean package -Pdist -DskipTests
+ }}}
+ 
+ {{{
+ export HADOOP_COMMON_HOME=$(pwd)/$(ls -d hadoop-common-project/hadoop-common/target/hadoop-common-*-SNAPSHOT)
+ export HADOOP_HDFS_HOME=$(pwd)/$(ls -d hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-*-SNAPSHOT)
+ export HADOOP_MAPRED_HOME=$(pwd)/$(ls -d hadoop-mapreduce-project/target/hadoop-mapreduce-*-SNAPSHOT)
+ export YARN_HOME=$HADOOP_MAPRED_HOME
+ export PATH=$HADOOP_COMMON_HOME/bin:$HADOOP_HDFS_HOME/bin:$HADOOP_MAPRED_HOME/bin:$PATH
+ }}}
+ 
+ Configure YARN to start a MR shuffle service:
+ 
+ {{{
+ cat > $YARN_HOME/conf/yarn-site.xml  << EOF
+ <?xml version="1.0"?>
+ <configuration>
+     <!-- Site specific YARN configuration properties -->
+     <property>
+       <name>yarn.nodemanager.aux-services</name>
+       <value>mapreduce.shuffle</value>
+     </property>
+     <property>
+       <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
+       <value>org.apache.hadoop.mapred.ShuffleHandler</value>
+     </property>
+ </configuration>
+ EOF
+ }}}
+ 
+ Run YARN daemons:
+ {{{
+ yarn resourcemanager
+ yarn nodemanager
+ }}}
+ 
+ Run a MR job:
+ {{{
+ cd hadoop-mapreduce-project
+ ant examples -Dresolvers=internal
+ cd ..
+ export HADOOP_CLASSPATH=$YARN_HOME/modules/*
+ mkdir in
+ cp BUILDING.txt in/
+ hadoop jar hadoop-mapreduce-project/build/hadoop-mapreduce-examples-*.jar wordcount -Dmapreduce.job.user.name=$USER
in out
+ }}}
+ 
  = Build Errors =
  
  == /code/hadoop-core-trunk/build.xml:634: Could not create task or type of type: junit.
==

Mime
View raw message