incubator-giraph-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ach...@apache.org
Subject svn commit: r1221461 - in /incubator/giraph/trunk: CHANGELOG README
Date Tue, 20 Dec 2011 19:53:47 GMT
Author: aching
Date: Tue Dec 20 19:53:47 2011
New Revision: 1221461

URL: http://svn.apache.org/viewvc?rev=1221461&view=rev
Log:
GIRAPH-110: Add guide to setup the enviroment for running the
unittests in a pseudo-distributed hadoop instance. (ssc via aching)


Modified:
    incubator/giraph/trunk/CHANGELOG
    incubator/giraph/trunk/README

Modified: incubator/giraph/trunk/CHANGELOG
URL: http://svn.apache.org/viewvc/incubator/giraph/trunk/CHANGELOG?rev=1221461&r1=1221460&r2=1221461&view=diff
==============================================================================
--- incubator/giraph/trunk/CHANGELOG (original)
+++ incubator/giraph/trunk/CHANGELOG Tue Dec 20 19:53:47 2011
@@ -2,6 +2,9 @@ Giraph Change Log
 
 Release 0.70.0 - unreleased
 
+  GIRAPH-110: Add guide to setup the enviroment for running the
+  unittests in a pseudo-distributed hadoop instance. (ssc via aching)
+
   GIRAPH-73: A little refactoring. (ssc via aching)
 
   GIRAPH-106: Change prepareSuperstep() to make

Modified: incubator/giraph/trunk/README
URL: http://svn.apache.org/viewvc/incubator/giraph/trunk/README?rev=1221461&r1=1221460&r2=1221461&view=diff
==============================================================================
--- incubator/giraph/trunk/README (original)
+++ incubator/giraph/trunk/README Tue Dec 20 19:53:47 2011
@@ -56,7 +56,8 @@ Building and testing:
 
 You will need the following:
 - Java 1.6
-- Maven 3 or higher. Giraph uses the munge plugin (http://sonatype.github.com/munge-maven-plugin/),
+- Maven 3 or higher. Giraph uses the munge plugin 
+  (http://sonatype.github.com/munge-maven-plugin/),
   which requires Maven 3, to support multiple versions of Hadoop. Also, the
   web site plugin requires Maven 3.
 
@@ -64,8 +65,6 @@ Use the maven commands with secure Hadoo
 - compile (i.e mvn compile)
 - package (i.e. mvn package)
 - test (i.e. mvn test)
--- For testing, one can submit the test to a running Hadoop instance
-   (i.e. mvn test -Dprop.mapred.job.tracker=localhost:50300)
 
 For the non-secure versions of Hadoop, run the maven commands with the
 additional argument '-Dhadoop=non_secure' to enable the maven profile
@@ -79,6 +78,51 @@ example compilation command is 'mvn -Dha
 -Dhadoop.jar.path=/tmp/hadoop-0.20.1-core.jar compile'.
 
 
+How to run the unittests on a local pseudo-distributed Hadoop instance:
+
+As mentioned earlier, Giraph supports several versions of Hadoop.  In
+this section, we describe how to run the Giraph unittests against a single
+node instance of Apache Hadoop 0.20.203.
+
+Download Apache Hadoop 0.20.203 (hadoop-0.20.203.0/hadoop-0.20.203.0rc1.tar.gz)
+from a mirror picked at http://www.apache.org/dyn/closer.cgi/hadoop/common/
+and unpack it into a local directory
+
+Follow the guide at 
+http://hadoop.apache.org/common/docs/r0.20.2/quickstart.html#PseudoDistributed
+to setup a pseudo-distributed single node Hadoop cluster.
+
+Giraph’s code assumes that you can run at least 4 mappers at once, 
+unfortunately the default configuration allows only 2. Therefore you need 
+to update conf/mapred-site.xml:
+
+<property>
+  <name>mapred.tasktracker.map.tasks.maximum</name>
+  <value>4</value>
+</property>
+
+<property>
+  <name>mapred.map.tasks</name>
+  <value>4</value>
+</property>
+
+After preparing the local filesystem with:
+
+rm -rf /tmp/hadoop-<username>
+/path/to/hadoop/bin/hadoop namenode -format
+
+you can start the local hadoop instance:
+
+/path/to/hadoop/bin/start-all.sh
+
+and finally run Giraph’s unittests:
+
+mvn clean test -Dprop.mapred.job.tracker=localhost:9001
+
+Now you can open a browser, point it to http://localhost:50030 and watch the
+Giraph jobs from the unittests running on your local Hadoop instance!
+
+
 Notes: 
 Counter limit: In Hadoop 0.20.203.0 onwards, there is a limit on the number of
 counters one can use, which is set to 120 by default. This limit restricts the



Mime
View raw message