hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Update of "HarmonyMapreduce" by GuillermoCabrera
Date Tue, 16 Nov 2010 17:21:13 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The "HarmonyMapreduce" page has been changed by GuillermoCabrera.
http://wiki.apache.org/hadoop/HarmonyMapreduce

--------------------------------------------------

New page:
= Hadoop hdfs on Harmony/Harmony Select 6 =

Similar to Hadoop common and hdfs, many of the steps outlined in this page could be skipped.
The build process is done without building native libraries as there are some issues we describe
below. Under scripts and patches below there are currently 21 failures for this project, out
of which 2 fail because of an explicit call that is made in the test case to sun/reflect from
the [[http://mockito.org/|mockito]] project.

=== Environment Setup ===

 1. Download missing dependencies: Apache Forrest, Xerces-C++ Parser, Apache Ant, Apache Maven,
Java JRE 5.0 (Needed by Apache Forrest)
 2. Download ecj jar from the [[http://download.eclipse.org/eclipse/downloads/drops/S-3.7M3-201010281441/index.php|3.7M3
build]] (this particular build contains a fix to bug affecting build process), place it in
`$HARMONY_HOME/hdk/jdk/lib` and add to bootclasspath.properties
 3. Add tools.jar from `$HARMONY_HOME/hdk/jdk/lib` to `$HARMONY_HOME/hdk/jdk/jre/lib/boot`.
Then, add entry of tools.jar into bootclasspath.properties in `$HARMONY_HOME/hdk/jdk/jre/lib/boot`
 4. Create a jar file (sun-javadoc.jar) in `$HARMONY_HOME/hdk/jdk/jre/lib/boot` containing
all javadoc related classes from SUN JDK 1.6. Then add entry of sun-javadoc.jar into bootclasspath.properties
in `$HARMONY_HOME/hdk/jdk/jre/lib/boot`
 5. Download Hadoop mapreduce {{{
 % svn checkout http://svn.apache.org/repos/asf/hadoop/mapreduce/tags/release-0.21.0/ mapreduce
}}}
 6. Download patches and place in appropriate directory (refer to script)
 7. Download, modify and run build script

=== Testing ===

 1. Copy swing.jar from Harmony 6 into Harmony Select's `jre/lib/ext` (Ivy requires swing
to run testing framework)
 2. Copy rmi.jar from Harmony 6 into Harmony Select's `jre/lib/boot` and add entry into `bootclasspath.properties`
 3. Comment out the line `xmlsec-1.4.3/commons-logging.jar` in Harmony Select's 'jre/lib/boot/bootclasspath.properties`
 4. Create a jar file (sunExtra.jar) in `$HARMONY_HOME/hdk/jdk/jre/lib/boot` containing all
the sun/reflect and sun/misc classes from SUN JDK 1.6. Then add entry of sunExtra.jar into
bootclasspath.properties
 5. Create soft link to libharmonyvm.so from libjvm.so in Harmony Select's `jre/bin/default`
 6. Download, modify and run test script

=== Patches ===

 * [[https://issues.apache.org/jira/browse/MAPREDUCE-2183| MAPREDUCE-2183]]
 * [[https://issues.apache.org/jira/browse/MAPREDUCE-2190| MAPREDUCE-2190]]

=== Build Script ===

{{{
# !/bin/sh
export SUBPROJECT=mapreduce
export VERSION=0.21.0
export VERSIONBUILD=0.21.0-SNAPSHOT
export PATCH_DIR=/home/harmony/Hadoop-Patches/Harmony/$VERSION-$SUBPROJECT
#Folder containing a clean version of Hadoop common needed to install patches
export PRISTINE=/home/harmony/Hadoop-Versions/pristine
#Note we are using SUN JDK, please refer below to the issues section and why we don't use
Harmony
export JAVA_HOME=/home/harmony/Java-Versions/jdk1.6.0_14
export HADOOP_INSTALL=/home/harmony/Hadoop-Versions/hadoop-$VERSION
export FORREST_INSTALL=/home/harmony/Test-Dependencies/apache-forrest-0.8
export XERCES_INSTALL=/home/harmony/Test-Dependencies/xerces-c_2_8_0
export ANT_HOME=/home/harmony/Test-Dependencies/apache-ant-1.8.1
#Java 5 required by Forrest
export JAVA5=/home/harmony/Java-Versions/ibm-java2-i386-50/jre
export PATH=$PATH:$ANT_HOME/bin
export CFLAGS=-m32
export CXXFLAGS=-m32
set PATH=$PATH:$JAVA_HOME

#clean (Clean targets are necessary to apply patches)
echo "Cleaning and Copying From Pristine"
rm -rf $HADOOP_INSTALL/$SUBPROJECT
cp -r $PRISTINE/hadoop-$VERSION/$SUBPROJECT $HADOOP_INSTALL/$SUBPROJECT

# Apply Patches
echo "Applying Patches"
cd $HADOOP_INSTALL/$SUBPROJECT

patch -p0 < $PATCH_DIR/MAPREDUCE-2183.patch
patch -p0 < $PATCH_DIR/MAPREDUCE-2190.patch

echo "Starting Bulid"
ant -Dversion=$VERSIONBUILD -Dcompile.c++=true -Dlibrecordio=true -Dxercescroot=$XERCES_INSTALL
-forrest.home=$FORREST_INSTALL -Djava5.home=$JAVA5 mvn-install -Dresolvers=internal > /home/harmony/Test-Scripts/Hadoop-$VERSION/SUN32build-$SUBPROJECT-noNative.out
2>&1
}}}

=== Test Script ===
The following script only runs (does not compile) all of the tests in the test-core target.
By running this script it is assumed that you already built Hadoop common with another JDK.
{{{
# !/bin/sh
export SUBPROJECT=mapreduce
export VERSION=0.21.0
export VERSIONBUILD=0.21.0-SNAPSHOT
export JAVA_HOME=/home/harmony/Java-Versions/harmonySelect6-1022137/java6/target/hdk/jdk
export HADOOP_INSTALL=/home/harmony/Hadoop-Versions/hadoop-$VERSION
export ANT_HOME=/home/harmony/Test-Dependencies/apache-ant-1.8.1
export PATH=$PATH:$ANT_HOME/bin
set PATH=$PATH:$JAVA_HOME

cd $HADOOP_INSTALL/$SUBPROJECT

echo "Testing Hadoop Common"
ant -Dsun.arch.data.model=32 -Dversion=$VERSIONBUILD run-test-core-nocompile -Dresolvers=internal
> /home/harmony/Test-Scripts/Hadoop-$VERSION/HSTest-$SUBPROJECT.out 2>&1
}}}

Note: To run a single test case, you need to add the -Dtestcase=testClass property to the
ant execution line.

=== Issues ===

==== Building Native ====

Building native libraries has been included in the build script above, thus, there is no need
to build using native, or copy native libraries as was done with Hadoop common and hdfs.
}}}

For further reference, please note the guidelines when trying to build native libraries, [[http://hadoop.apache.org/common/docs/current/native_libraries.html|Native
Libraries Guide]]

==== Others ====

 * Mapreduce does not compile when using Apache Harmony 6. There is 1 error in this project
in the SpillRecord.java file (line 141) where we get the message "The type IndexRecord is
already defined". Initial guess is that this is related to ecj as we had a similar problem
with primary types before. For this reason we are using SUN to build mapreduce.

Mime
View raw message