Mailing-List: contact common-commits-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-dev@hadoop.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
From: Apache Wiki <wikidiffs@apache.org>
To: Apache Wiki <wikidiffs@apache.org>
Date: Tue, 16 Nov 2010 17:21:13 -0000
Message-ID: <20101116172113.13049.87533@eosnew.apache.org>
Subject: 
 =?utf-8?q?=5BHadoop_Wiki=5D_Update_of_=22HarmonyMapreduce=22_by_Guillermo?=
 =?utf-8?q?Cabrera?=

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for ch=
ange notification.

The "HarmonyMapreduce" page has been changed by GuillermoCabrera.
http://wiki.apache.org/hadoop/HarmonyMapreduce

--------------------------------------------------

New page:
=3D Hadoop hdfs on Harmony/Harmony Select 6 =3D

Similar to Hadoop common and hdfs, many of the steps outlined in this page =
could be skipped. The build process is done without building native librari=
es as there are some issues we describe below. Under scripts and patches be=
low there are currently 21 failures for this project, out of which 2 fail b=
ecause of an explicit call that is made in the test case to sun/reflect fro=
m the [[http://mockito.org/|mockito]] project.

=3D=3D=3D Environment Setup =3D=3D=3D

 1. Download missing dependencies: Apache Forrest, Xerces-C++ Parser, Apach=
e Ant, Apache Maven, Java JRE 5.0 (Needed by Apache Forrest)
 2. Download ecj jar from the [[http://download.eclipse.org/eclipse/downloa=
ds/drops/S-3.7M3-201010281441/index.php|3.7M3 build]] (this particular buil=
d contains a fix to bug affecting build process), place it in `$HARMONY_HOM=
E/hdk/jdk/lib` and add to bootclasspath.properties
 3. Add tools.jar from `$HARMONY_HOME/hdk/jdk/lib` to `$HARMONY_HOME/hdk/jd=
k/jre/lib/boot`. Then, add entry of tools.jar into bootclasspath.properties=
 in `$HARMONY_HOME/hdk/jdk/jre/lib/boot`
 4. Create a jar file (sun-javadoc.jar) in `$HARMONY_HOME/hdk/jdk/jre/lib/b=
oot` containing all javadoc related classes from SUN JDK 1.6. Then add entr=
y of sun-javadoc.jar into bootclasspath.properties in `$HARMONY_HOME/hdk/jd=
k/jre/lib/boot`
 5. Download Hadoop mapreduce {{{
 % svn checkout http://svn.apache.org/repos/asf/hadoop/mapreduce/tags/relea=
se-0.21.0/ mapreduce
}}}
 6. Download patches and place in appropriate directory (refer to script)
 7. Download, modify and run build script

=3D=3D=3D Testing =3D=3D=3D

 1. Copy swing.jar from Harmony 6 into Harmony Select's `jre/lib/ext` (Ivy =
requires swing to run testing framework)
 2. Copy rmi.jar from Harmony 6 into Harmony Select's `jre/lib/boot` and ad=
d entry into `bootclasspath.properties`
 3. Comment out the line `xmlsec-1.4.3/commons-logging.jar` in Harmony Sele=
ct's 'jre/lib/boot/bootclasspath.properties`
 4. Create a jar file (sunExtra.jar) in `$HARMONY_HOME/hdk/jdk/jre/lib/boot=
` containing all the sun/reflect and sun/misc classes from SUN JDK 1.6. The=
n add entry of sunExtra.jar into bootclasspath.properties
 5. Create soft link to libharmonyvm.so from libjvm.so in Harmony Select's =
`jre/bin/default`
 6. Download, modify and run test script

=3D=3D=3D Patches =3D=3D=3D

 * [[https://issues.apache.org/jira/browse/MAPREDUCE-2183| MAPREDUCE-2183]]
 * [[https://issues.apache.org/jira/browse/MAPREDUCE-2190| MAPREDUCE-2190]]

=3D=3D=3D Build Script =3D=3D=3D

{{{
# !/bin/sh
export SUBPROJECT=3Dmapreduce
export VERSION=3D0.21.0
export VERSIONBUILD=3D0.21.0-SNAPSHOT
export PATCH_DIR=3D/home/harmony/Hadoop-Patches/Harmony/$VERSION-$SUBPROJECT
#Folder containing a clean version of Hadoop common needed to install patch=
es
export PRISTINE=3D/home/harmony/Hadoop-Versions/pristine
#Note we are using SUN JDK, please refer below to the issues section and wh=
y we don't use Harmony
export JAVA_HOME=3D/home/harmony/Java-Versions/jdk1.6.0_14
export HADOOP_INSTALL=3D/home/harmony/Hadoop-Versions/hadoop-$VERSION
export FORREST_INSTALL=3D/home/harmony/Test-Dependencies/apache-forrest-0.8
export XERCES_INSTALL=3D/home/harmony/Test-Dependencies/xerces-c_2_8_0
export ANT_HOME=3D/home/harmony/Test-Dependencies/apache-ant-1.8.1
#Java 5 required by Forrest
export JAVA5=3D/home/harmony/Java-Versions/ibm-java2-i386-50/jre
export PATH=3D$PATH:$ANT_HOME/bin
export CFLAGS=3D-m32
export CXXFLAGS=3D-m32
set PATH=3D$PATH:$JAVA_HOME

#clean (Clean targets are necessary to apply patches)
echo "Cleaning and Copying From Pristine"
rm -rf $HADOOP_INSTALL/$SUBPROJECT
cp -r $PRISTINE/hadoop-$VERSION/$SUBPROJECT $HADOOP_INSTALL/$SUBPROJECT

# Apply Patches
echo "Applying Patches"
cd $HADOOP_INSTALL/$SUBPROJECT

patch -p0 < $PATCH_DIR/MAPREDUCE-2183.patch
patch -p0 < $PATCH_DIR/MAPREDUCE-2190.patch

echo "Starting Bulid"
ant -Dversion=3D$VERSIONBUILD -Dcompile.c++=3Dtrue -Dlibrecordio=3Dtrue -Dx=
ercescroot=3D$XERCES_INSTALL -forrest.home=3D$FORREST_INSTALL -Djava5.home=
=3D$JAVA5 mvn-install -Dresolvers=3Dinternal > /home/harmony/Test-Scripts/H=
adoop-$VERSION/SUN32build-$SUBPROJECT-noNative.out 2>&1
}}}

=3D=3D=3D Test Script =3D=3D=3D
The following script only runs (does not compile) all of the tests in the t=
est-core target. By running this script it is assumed that you already buil=
t Hadoop common with another JDK.
{{{
# !/bin/sh
export SUBPROJECT=3Dmapreduce
export VERSION=3D0.21.0
export VERSIONBUILD=3D0.21.0-SNAPSHOT
export JAVA_HOME=3D/home/harmony/Java-Versions/harmonySelect6-1022137/java6=
/target/hdk/jdk
export HADOOP_INSTALL=3D/home/harmony/Hadoop-Versions/hadoop-$VERSION
export ANT_HOME=3D/home/harmony/Test-Dependencies/apache-ant-1.8.1
export PATH=3D$PATH:$ANT_HOME/bin
set PATH=3D$PATH:$JAVA_HOME

cd $HADOOP_INSTALL/$SUBPROJECT

echo "Testing Hadoop Common"
ant -Dsun.arch.data.model=3D32 -Dversion=3D$VERSIONBUILD run-test-core-noco=
mpile -Dresolvers=3Dinternal > /home/harmony/Test-Scripts/Hadoop-$VERSION/H=
STest-$SUBPROJECT.out 2>&1
}}}

Note: To run a single test case, you need to add the -Dtestcase=3DtestClass=
 property to the ant execution line.

=3D=3D=3D Issues =3D=3D=3D

=3D=3D=3D=3D Building Native =3D=3D=3D=3D

Building native libraries has been included in the build script above, thus=
, there is no need to build using native, or copy native libraries as was d=
one with Hadoop common and hdfs.
}}}

For further reference, please note the guidelines when trying to build nati=
ve libraries, [[http://hadoop.apache.org/common/docs/current/native_librari=
es.html|Native Libraries Guide]]

=3D=3D=3D=3D Others =3D=3D=3D=3D

 * Mapreduce does not compile when using Apache Harmony 6. There is 1 error=
 in this project in the SpillRecord.java file (line 141) where we get the m=
essage "The type IndexRecord is already defined". Initial guess is that thi=
s is related to ecj as we had a similar problem with primary types before. =
For this reason we are using SUN to build mapreduce.