Dear Wiki user,
You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.
The "HowToSetupYourDevelopmentEnvironment" page has been changed by TomWhite:
http://wiki.apache.org/hadoop/HowToSetupYourDevelopmentEnvironment?action=diff&rev1=29&rev2=30
Note that the {{{start-dfs.sh}}} script will not work with this set up, since it assumes
that HADOOP_COMMON_HOME and HADOOP_HDFS_HOME are the same directory.
+ = Run MapReduce in pseudo-distributed mode from the dev tree =
+
+ Build the packaging from the top level. This will build the distribution in an exploded
format that we can run directly (i.e. no need to untar):
+ {{{
+ mvn clean package -Pdist -DskipTests
+ }}}
+
+ {{{
+ export HADOOP_COMMON_HOME=$(pwd)/$(ls -d hadoop-common-project/hadoop-common/target/hadoop-common-*-SNAPSHOT)
+ export HADOOP_HDFS_HOME=$(pwd)/$(ls -d hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-*-SNAPSHOT)
+ export HADOOP_MAPRED_HOME=$(pwd)/$(ls -d hadoop-mapreduce-project/target/hadoop-mapreduce-*-SNAPSHOT)
+ export YARN_HOME=$HADOOP_MAPRED_HOME
+ export PATH=$HADOOP_COMMON_HOME/bin:$HADOOP_HDFS_HOME/bin:$HADOOP_MAPRED_HOME/bin:$PATH
+ }}}
+
+ Configure YARN to start a MR shuffle service:
+
+ {{{
+ cat > $YARN_HOME/conf/yarn-site.xml << EOF
+ <?xml version="1.0"?>
+ <configuration>
+ <!-- Site specific YARN configuration properties -->
+ <property>
+ <name>yarn.nodemanager.aux-services</name>
+ <value>mapreduce.shuffle</value>
+ </property>
+ <property>
+ <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
+ <value>org.apache.hadoop.mapred.ShuffleHandler</value>
+ </property>
+ </configuration>
+ EOF
+ }}}
+
+ Run YARN daemons:
+ {{{
+ yarn resourcemanager
+ yarn nodemanager
+ }}}
+
+ Run a MR job:
+ {{{
+ cd hadoop-mapreduce-project
+ ant examples -Dresolvers=internal
+ cd ..
+ export HADOOP_CLASSPATH=$YARN_HOME/modules/*
+ mkdir in
+ cp BUILDING.txt in/
+ hadoop jar hadoop-mapreduce-project/build/hadoop-mapreduce-examples-*.jar wordcount -Dmapreduce.job.user.name=$USER
in out
+ }}}
+
= Build Errors =
== /code/hadoop-core-trunk/build.xml:634: Could not create task or type of type: junit.
==
|