hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Update of "HarmonyCommon" by GuillermoCabrera
Date Wed, 10 Nov 2010 23:42:55 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The "HarmonyCommon" page has been changed by GuillermoCabrera.


New page:
= Hadoop Common =

Given that Harmony Select has a reduced subset of packages as compared with Harmony, we decided
to build using Harmony and then test using a Harmony Select JRE. The build process is done
without building native libraries as there are some issues we describe below. Under scripts
and patches below there are currently 12 failures for this project, out of which 5 fail because
of an explicit call that is made in the test case to sun/reflect from the [[http://mockito.org/|mockito]]

=== Environment Setup ===

 1. Download missing dependencies: Apache Forrest, Xerces-C++ Parser, Apache Ant, Apache Maven,
Java JRE 5.0 (Needed by Apache Forrest)
 2. Download ecj jar from the [[http://download.eclipse.org/eclipse/downloads/drops/S-3.7M3-201010281441/index.php|3.7M3
build]] (this particular build contains a fix to bug affecting build process), place it in
`$HARMONY_HOME/hdk/jdk/lib` and add to bootclasspath.properties
 3. Add tools.jar from `$HARMONY_HOME/hdk/jdk/lib` to `$HARMONY_HOME/hdk/jdk/jre/lib/boot`.
Then, add entry of tools.jar into bootclasspath.properties in `$HARMONY_HOME/hdk/jdk/jre/lib/boot`
 4. Create a jar file (sun-javadoc.jar) in `$HARMONY_HOME/hdk/jdk/jre/lib/boot` containing
all javadoc related classes from SUN JDK 1.6. Then add entry of sun-javadoc.jar into bootclasspath.properties
in `$HARMONY_HOME/hdk/jdk/jre/lib/boot`
 5. Download Hadoop common {{{
 % svn checkout http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.21.0/ common

 6. Download patches and place in appropriate directory (refer to script)
 7. Download, modify and run build script

=== Testing ===

 1. Copy swing.jar from Harmony 6 into Harmony Select's `jre/lib/ext` (Ivy requires swing
to run testing framework)
 2. Copy rmi.jar from Harmony 6 into Harmony Select's `jre/lib/boot` and add entry into `bootclasspath.properties`
 3. Comment out the line `xmlsec-1.4.3/commons-logging.jar` in Harmony Select's 'jre/lib/boot/bootclasspath.properties`
 4. Copy native libraries (libhadoop.*) from Apache Hadoop 0.21.0 release into `common/build/native/Linux-x86-32/lib`
 5. Create a jar file (sunExtra.jar) in `$HARMONY_HOME/hdk/jdk/jre/lib/boot` containing all
the sun/reflect and sun/misc classes from SUN JDK 1.6. Then add entry of sunExtra.jar into
 6. Create soft link to libharmonyvm.so from libjvm.so in Harmony Select's `jre/bin/default`
 7. Download, modify and run test script

=== Patches ===

 * [[https://issues.apache.org/jira/browse/HADOOP-6924| HADOOP-6924]]
 * [[https://issues.apache.org/jira/browse/HADOOP-6954| HADOOP-6952]]
 * [[https://issues.apache.org/jira/browse/HADOOP-6895| HADOOP-6895]]
 * [[https://issues.apache.org/jira/browse/HADOOP-7026| HADOOP-7026]]
 * [[https://issues.apache.org/jira/browse/HADOOP-7027| HADOOP-7027]]

=== Build Script ===

# !/bin/sh
export SUBPROJECT=common
export VERSION=0.21.0
export PATCH_DIR=/home/harmony/Hadoop-Patches/Harmony/$VERSION-$SUBPROJECT
#Folder containing a clean version of Hadoop common needed to install patches
export PRISTINE=/home/harmony/Hadoop-Versions/pristine
export JAVA_HOME=/home/harmony/Java-Versions/harmony6-1022137/java6/target/hdk/jdk
export HADOOP_INSTALL=/home/harmony/Hadoop-Versions/hadoop-$VERSION
export FORREST_INSTALL=/home/harmony/Test-Dependencies/apache-forrest-0.8
export XERCES_INSTALL=/home/harmony/Test-Dependencies/xerces-c_2_8_0
export ANT_HOME=/home/harmony/Test-Dependencies/apache-ant-1.8.1
#Java 5 required by Forrest
export JAVA5=/home/harmony/Java-Versions/ibm-java2-i386-50/jre
export PATH=$PATH:$ANT_HOME/bin

#clean (Clean targets are necessary to apply patches)
echo "Cleaning and Copying From Pristine"

# Apply Patches
echo "Applying Patches"
#The following test case produces an error that halts the testing process
rm src/test/core/org/apache/hadoop/util/TestReflectionUtils.java

patch -p0 < $PATCH_DIR/HADOOP-6895.patch
patch -p0 < $PATCH_DIR/HADOOP-6924-v2.patch
patch -p0 < $PATCH_DIR/HADOOP-6954.patch

# Clean, Build and Run the Core (Non-Contrib) Unit Tests
echo "Starting Building and Unit Test"
ant -Dversion=$VERSIONBUILD -Dxercescroot=$XERCES_INSTALL -Dforrest.home=$FORREST_INSTALL
-Djava5.home=$JAVA5 clean mvn-install > /home/harmony/Test-Scripts/Hadoop-$VERSION/Harmonybuild-$SUBPROJECT-noNative.out

=== Test Script ===
The following script only runs (does not compile) all of the tests in the test-core target.
By running this script it is assumed that you already built Hadoop common with another JDK.
# !/bin/sh
export SUBPROJECT=common
export VERSION=0.21.0
export JAVA_HOME=/home/harmony/Java-Versions/harmonySelect6-1022137/java6/target/hdk/jdk/jre
export HADOOP_INSTALL=/home/harmony/Hadoop-Versions/hadoop-$VERSION
export ANT_HOME=/home/harmony/Test-Dependencies/apache-ant-1.8.1
export PATH=$PATH:$ANT_HOME/bin


echo "Testing Hadoop Common"
ant -Dsun.arch.data.model=32 -Dversion=$VERSIONBUILD -Dresolvers=internal run-test-core-nocompile
> /home/harmony/Test-Scripts/Hadoop-$VERSION/HSTest-$SUBPROJECT.out 2>&1


Note: To run a single test case, you need to add the -Dtestcase=testClass property to the
ant execution line.

=== Issues ===

==== Building Native ====

To build using native libraries, you need the following additions to your build script:
export CFLAGS=-m32
export CXXFLAGS=-m32

Also, please note the guidelines when trying to build native libraries, [[http://hadoop.apache.org/common/docs/current/native_libraries.html|Native
Libraries Guide]]

 * javah is not available in Harmony 6. You need to copy tools.jar from Sun into Harmony and
add to the bootclasspath.properties file
 * Even with the created soft link (libjvm.so) to libharmonyvm.so there are still issues where
the configure script in `common/src/native` cannont find -ljvm.

==== Others ====

 * There are two test cases (`TestArrayFile` and `TestMapFile`) that crash during testing,
but succeed when run individually.
 * TestSaslRPC reports `Unable to find SASL client implementation`

View raw message