Mailing-List: contact common-commits-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-dev@hadoop.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
From: Apache Wiki <wikidiffs@apache.org>
To: Apache Wiki <wikidiffs@apache.org>
Date: Wed, 10 Nov 2010 23:42:55 -0000
Message-ID: <20101110234255.12007.40121@eosnew.apache.org>
Subject: 
 =?utf-8?q?=5BHadoop_Wiki=5D_Update_of_=22HarmonyCommon=22_by_GuillermoCab?=
 =?utf-8?q?rera?=

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for ch=
ange notification.

The "HarmonyCommon" page has been changed by GuillermoCabrera.
http://wiki.apache.org/hadoop/HarmonyCommon

--------------------------------------------------

New page:
=3D Hadoop Common =3D

Given that Harmony Select has a reduced subset of packages as compared with=
 Harmony, we decided to build using Harmony and then test using a Harmony S=
elect JRE. The build process is done without building native libraries as t=
here are some issues we describe below. Under scripts and patches below the=
re are currently 12 failures for this project, out of which 5 fail because =
of an explicit call that is made in the test case to sun/reflect from the [=
[http://mockito.org/|mockito]] project.

=3D=3D=3D Environment Setup =3D=3D=3D

 1. Download missing dependencies: Apache Forrest, Xerces-C++ Parser, Apach=
e Ant, Apache Maven, Java JRE 5.0 (Needed by Apache Forrest)
 2. Download ecj jar from the [[http://download.eclipse.org/eclipse/downloa=
ds/drops/S-3.7M3-201010281441/index.php|3.7M3 build]] (this particular buil=
d contains a fix to bug affecting build process), place it in `$HARMONY_HOM=
E/hdk/jdk/lib` and add to bootclasspath.properties
 3. Add tools.jar from `$HARMONY_HOME/hdk/jdk/lib` to `$HARMONY_HOME/hdk/jd=
k/jre/lib/boot`. Then, add entry of tools.jar into bootclasspath.properties=
 in `$HARMONY_HOME/hdk/jdk/jre/lib/boot`
 4. Create a jar file (sun-javadoc.jar) in `$HARMONY_HOME/hdk/jdk/jre/lib/b=
oot` containing all javadoc related classes from SUN JDK 1.6. Then add entr=
y of sun-javadoc.jar into bootclasspath.properties in `$HARMONY_HOME/hdk/jd=
k/jre/lib/boot`
 5. Download Hadoop common {{{
 % svn checkout http://svn.apache.org/repos/asf/hadoop/common/tags/release-=
0.21.0/ common =

}}}
 6. Download patches and place in appropriate directory (refer to script)
 7. Download, modify and run build script

=3D=3D=3D Testing =3D=3D=3D

 1. Copy swing.jar from Harmony 6 into Harmony Select's `jre/lib/ext` (Ivy =
requires swing to run testing framework)
 2. Copy rmi.jar from Harmony 6 into Harmony Select's `jre/lib/boot` and ad=
d entry into `bootclasspath.properties`
 3. Comment out the line `xmlsec-1.4.3/commons-logging.jar` in Harmony Sele=
ct's 'jre/lib/boot/bootclasspath.properties`
 4. Copy native libraries (libhadoop.*) from Apache Hadoop 0.21.0 release i=
nto `common/build/native/Linux-x86-32/lib`
 5. Create a jar file (sunExtra.jar) in `$HARMONY_HOME/hdk/jdk/jre/lib/boot=
` containing all the sun/reflect and sun/misc classes from SUN JDK 1.6. The=
n add entry of sunExtra.jar into bootclasspath.properties
 6. Create soft link to libharmonyvm.so from libjvm.so in Harmony Select's =
`jre/bin/default`
 7. Download, modify and run test script

=3D=3D=3D Patches =3D=3D=3D

 * [[https://issues.apache.org/jira/browse/HADOOP-6924| HADOOP-6924]]
 * [[https://issues.apache.org/jira/browse/HADOOP-6954| HADOOP-6952]]
 * [[https://issues.apache.org/jira/browse/HADOOP-6895| HADOOP-6895]]
 * [[https://issues.apache.org/jira/browse/HADOOP-7026| HADOOP-7026]]
 * [[https://issues.apache.org/jira/browse/HADOOP-7027| HADOOP-7027]]

=3D=3D=3D Build Script =3D=3D=3D

{{{
# !/bin/sh
export SUBPROJECT=3Dcommon
export VERSION=3D0.21.0
export VERSIONBUILD=3D0.21.0-SNAPSHOT
export PATCH_DIR=3D/home/harmony/Hadoop-Patches/Harmony/$VERSION-$SUBPROJECT
#Folder containing a clean version of Hadoop common needed to install patch=
es
export PRISTINE=3D/home/harmony/Hadoop-Versions/pristine
export JAVA_HOME=3D/home/harmony/Java-Versions/harmony6-1022137/java6/targe=
t/hdk/jdk
export HADOOP_INSTALL=3D/home/harmony/Hadoop-Versions/hadoop-$VERSION
export FORREST_INSTALL=3D/home/harmony/Test-Dependencies/apache-forrest-0.8
export XERCES_INSTALL=3D/home/harmony/Test-Dependencies/xerces-c_2_8_0
export ANT_HOME=3D/home/harmony/Test-Dependencies/apache-ant-1.8.1
#Java 5 required by Forrest
export JAVA5=3D/home/harmony/Java-Versions/ibm-java2-i386-50/jre
export PATH=3D$PATH:$ANT_HOME/bin
set PATH=3D$PATH:$JAVA_HOME

#clean (Clean targets are necessary to apply patches)
echo "Cleaning and Copying From Pristine"
rm -rf $HADOOP_INSTALL/$SUBPROJECT
cp -r $PRISTINE/hadoop-$VERSION/$SUBPROJECT $HADOOP_INSTALL/$SUBPROJECT

# Apply Patches
echo "Applying Patches"
cd $HADOOP_INSTALL/$SUBPROJECT
#The following test case produces an error that halts the testing process
rm src/test/core/org/apache/hadoop/util/TestReflectionUtils.java

patch -p0 < $PATCH_DIR/HADOOP-6895.patch
patch -p0 < $PATCH_DIR/HADOOP-6924-v2.patch
patch -p0 < $PATCH_DIR/HADOOP-6954.patch
patch -p0 < $PATCH_DIR/HADOOP-UserFile.patchXXXXXXXXXXXXXXXXXXXX
patch -p0 < $PATCH_DIR/HADOOP-COMMON-Targets.patchXXXXXXXXXXXXXXXXXXXXX

# Clean, Build and Run the Core (Non-Contrib) Unit Tests
echo "Starting Building and Unit Test"
ant -Dversion=3D$VERSIONBUILD -Dxercescroot=3D$XERCES_INSTALL -Dforrest.hom=
e=3D$FORREST_INSTALL -Djava5.home=3D$JAVA5 clean mvn-install > /home/harmon=
y/Test-Scripts/Hadoop-$VERSION/Harmonybuild-$SUBPROJECT-noNative.out 2>&1
}}}

=3D=3D=3D Test Script =3D=3D=3D
The following script only runs (does not compile) all of the tests in the t=
est-core target. By running this script it is assumed that you already buil=
t Hadoop common with another JDK.
{{{
# !/bin/sh
export SUBPROJECT=3Dcommon
export VERSION=3D0.21.0
export VERSIONBUILD=3D0.21.0-SNAPSHOT
export JAVA_HOME=3D/home/harmony/Java-Versions/harmonySelect6-1022137/java6=
/target/hdk/jdk/jre
export HADOOP_INSTALL=3D/home/harmony/Hadoop-Versions/hadoop-$VERSION
export ANT_HOME=3D/home/harmony/Test-Dependencies/apache-ant-1.8.1
export PATH=3D$PATH:$ANT_HOME/bin
set PATH=3D$PATH:$JAVA_HOME

cd $HADOOP_INSTALL/$SUBPROJECT

echo "Testing Hadoop Common"
ant -Dsun.arch.data.model=3D32 -Dversion=3D$VERSIONBUILD -Dresolvers=3Dinte=
rnal run-test-core-nocompile > /home/harmony/Test-Scripts/Hadoop-$VERSION/H=
STest-$SUBPROJECT.out 2>&1

}}}

Note: To run a single test case, you need to add the -Dtestcase=3DtestClass=
 property to the ant execution line.

=3D=3D=3D Issues =3D=3D=3D

=3D=3D=3D=3D Building Native =3D=3D=3D=3D

To build using native libraries, you need the following additions to your b=
uild script:
{{{
export CFLAGS=3D-m32
export CXXFLAGS=3D-m32
-Dcompile.native=3Dtruek
}}}

Also, please note the guidelines when trying to build native libraries, [[h=
ttp://hadoop.apache.org/common/docs/current/native_libraries.html|Native Li=
braries Guide]]

 * javah is not available in Harmony 6. You need to copy tools.jar from Sun=
 into Harmony and add to the bootclasspath.properties file
 * Even with the created soft link (libjvm.so) to libharmonyvm.so there are=
 still issues where the configure script in `common/src/native` cannont fin=
d -ljvm.

=3D=3D=3D=3D Others =3D=3D=3D=3D

 * There are two test cases (`TestArrayFile` and `TestMapFile`) that crash =
during testing, but succeed when run individually.
 * TestSaslRPC reports `Unable to find SASL client implementation`