mahout-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject [CONF] Apache Mahout > BuildingMahout
Date Mon, 11 Jul 2011 22:06:00 GMT
Space: Apache Mahout (
Page: BuildingMahout (

Edited by Marc Millstone:
h1. Prerequisites for Building Mahout

* Java JDK *1.6*
* Maven *2.0.11* or higher ([])

h1. Get the Source Code

h2. Latest (Recommended)

Use [Subversion|] to check out the code:
svn co

h2. Release

[Download source |]
Maven artifacts should be in the usual place: []

h1. Compiling

* change directory to the checked out directory
* mvn install


If you are Compiling under Windows, make sure you installed Cygwin correctly. Here is a [good
tutorial|] on installing and configuring
a Hadoop cluster on Windows, and it points out at antoher great tutorial about installing
Cygwin. Here is another [good tutorial |]
for setting up Hadoop on Windows (via Cygwin) along with the corresponding Eclipse plugin
for easier Map-Reduce development and deployment.
Also if your Windows' Account name contains spaces (for example 'my account') some of the
tests wont pass and the build will fail.
The easiest solution is to create a new Windows' Account that contains no spaces (for example
'myaccount'), and use that account when Compiling.
{note}This will run the default targets, which builds both the core and the examples, and
also packages them.

h2. Compile Core

* change to the core directory
* mvn compile

h2. Install Core

Note, you can do install instead of compile.

* change to the core directory
* mvn install

h2. Compile Examples

*You must "mvn install" the core before you can build the examples.  For some reason Maven
doesn't know how to build sibling modules that are dependencies.*

* change to the examples directory
* mvn compile

h2. Compile Taste Web

* change to the taste-web directory
* Edit the value to add in your recommender class.  Make sure the recommender
class is available in the classpath (i.e. add it to the WAR that gets created)
* mvn package

h3. Adding your own Taste recommender

Now MAHOUT-110 has been committed, add your recommender JAR file into trunk/taste-web/lib
and then edit the file and set the recommender.class property to the
fully qualified name of your Recommender.

Then, doing a "mvn package" will bake your JAR file into the WAR file by adding it to WEB-INF/lib
and setting the file will automatically configure the web.xml to use

h3. Deploying Taste Web

Instructions for deploying and getting a taste of Taste Web can be found at the [documentation

h2. Working With Maven in Eclipse {anchor:mahout_maven_eclipse}

We've used Eclipse Galileo and m2eclipse 0.9 and the 'import maven projects' feature. Check
out the mahout sources into your workspace directory, do a full build on the command-line
and then fire up the import in Eclipse from File > Import > Maven Projects. Point it
at the mahout root directory. You are then given the opportunity to choose which sub-modules
to import. You don't need to import them all, only the projects you are interested in working

This sets up one Eclipse project for each of the mahout sub-modules you chose. Inter-project
dependencies are automatically resolved. For example, if mahout-core and mahout-math are both
open the m2eclipse plugin will automatically set up a project dependency on mahout-math in
mahout-core. If you close mahout-math, the plugin will automatically revert to a jar dependency
for mahout-math.

If you are importing mahout-collections/mahout-math you will have to add the target/generated-sources
directories to your build path manually and do a refresh on the dependent projects. Alternatively
just avoid importing these (or close them) and they will be treated as a regular jar dependency.
This works much better than doing the checkout into Eclipse directly via the m2eclipse 'check
out maven projects from scm' importer.

h2. For Eclipse users on Mac OS X Leopard

These instructions work on  Mac OSX Leopard 10.5.6 and Eclipse 3.3.2
# Get the [source code|#Get the Source Code]. You can use [Subclipse plugin|]
for Eclipse
# Install Maven plugin for eclipse through the update site present in [M2Eclipse|]
# JDK 1.6
Since hadoop requires jdk 1.6, Mahout also needs jdk 1.6 and you would have to make sure JRE
1.6 is added to Eclipse.
To use JRE 1.6 for Mahout, go to Preferences --> Java --> Installed JREs \--> Click
Add and specify
## JRE type as "standard VM"
## JRE Name as "JVM 1.6" and
## JRE home directory as /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home and
click Ok
## Select this new JVM (JVM 1.6) and click Ok
# Building Mahout - Right-click on the mahout project and choose "run as" \--> "maven project"

h2. Common problems building Mahout

Sometimes the compilation may fail. Depending on the error type the tips below may help.

* Type 1: "artifact not found, please download and install it manually" \--> clean maven
repository and try again
rm -rf ~/.m2/

* Type 2: "build failed because of a single test - 1 test failed, Failures: 1"  \--> retry
compilation from clean (you may need to clean the \~/.m2 directory above)
mvn clean install

* Type 3 (MacOSX only): Wrong Java Version

Problem: There is an error 'javac: invalid target release: 1.6' even though Java 6 is set
to be the default in the Java Preferences. Even on the command line, 'java \-version' showed
1.6 as the version number. However, this did not carry over to Maven, as 'mvn \-v' confirmed.

Solution: Explicitly set the 'JAVA_HOME' environment variable. Strangely enough, this does
not happen automatically when changing the Java Preferences.  In my case, I set it via 'export

If you don't want to hard-code which java version you use, you can use this:

export JAVA_HOME=$(/usr/libexec/java_home)

This makes use of whatever version of Java you have set on the Java control panel.

* Type 4: Out of Memory when compiling

Problem: 'java.lang.OutOfMemoryError: Java heap space' when compiling the core module of a
current svn checkout of Mahout (not the release).

Solution: Set the environment variable 'MAVEN_OPTS' to allow for more memory via 'export MAVEN_OPTS=-Xmx1024m'

h2. Building Mahout with the IBM JVM
Due to Hadoop using some Sun proprietary API's in version (the
version of Hadoop used by version of Mahout in trunk), some care must be
taken when building Mahout from source. 

To build Mahout: 

# using [Subversion|], check out the trunk:
svn co
Using trunk is required due to a failing unit test under the IBM JVM in Mahout .5.

# Set the environment variable JAVA_HOME to the correct location.
# Update the pom.xml in the project root directory to use Hadoop version 0.20.2 by 
 setting the version number for the Hadoop version to the following:
       <version> 0.20.2</version>  
# mvn install will now build and test your Mahout distribution.

h2. Some tips to speed up installation by skipping testing

Although we love testing and running the tests is strongly recommended for the first installation,
there are instances when you may want to skip testing altogether, or only include a subset
of them. This can shorten the lengthy installation time of nearly 15 minutes to a matter of
10 to 15 seconds. Assuming MAHOUT_HOME is the directory of Mahout, to achieve fast installation
you have the following options depending on your requirements:

h3. Avoid testing completely.

This is useful when you've made changes to files which are NOT under the tests directory (MAHOUT_HOME/core/src/test/...
or MAHOUT_HOME/math/src/test/...), or you have added new code under MAHOUT_HOME/core/src/main/
or MAHOUT_HOME/math/src/main/ and want to see if it compiles. Change to the MAHOUT_HOME directory,
and type:

mvn -DskipTests install

This will compile and install Mahout's classes, including your new code, and skip ALL tests.

h3. Running only a few tests: Method I

Let's say you have implemented a new feature by making changes to a source file called MyNewFeature,
and you have written some corresponding unit tests for it in the file called "TestMyNewFeature"
under the tests directory. To only run the tests of TestMyNewFeature class, from the MAHOUT_HOME
directory, type:

mvn -Dtest=TestMyNewFeature install

The TestMyNewFeature class should be passed as an argument to Maven's mvn command without
any path information. Just the class name is needed.

To run multiple test classes, you can do:

mvn -Dtest=TestMyNewFeature,TestAnotherNewFeature install

h3. Running only a few tests: Method II

The pom.xml file present in the MAHOUT_HOME directory contains all the information needed
by the mvn (Maven) command to compile, test, install, package etc. It is one place from where
you can control testing as well. Maven uses the Surefire plugin to run JUnit tests, so to
modify the default testing behavior of running all tests, you can modify the pom.xml to <include>
only the tests of TestMyNewFeature class, which was used as an example above. Open the pom.xml
present in MAHOUT_HOME in your favorite editor, and find the following lines:

          <argLine>-Xms256m -Xmx512m</argLine>

Modify these lines to include only the TestMyNewFeature class while testing, by using the
<includes> and <include> tags:

          <argLine>-Xms256m -Xmx512m</argLine>

Next, save the modified pom.xml file and from the MAHOUT_HOME directory type:

mvn install

This will only run the tests in the TestMyNewFeature class and install Mahout for you. Note
that now you don't have to mention -Dtest=TestMyNewFeature on the command line.


Change your notification preferences:

View raw message