Native Hadoop Libraries

Purpose

Hadoop has native implementations of certain components for reasons of + both performace & non-availability of Java implementations. These + components are available in a single, dynamically-linked, native library. + On the *nix platform it is libhadoop.so. This document describes + the usage & details on how to build the native libraries.

Components

Hadoop currently has the following + + compression codecs as the native components:

+zlib +
+gzip +
+lzo +

Of the above, the availability of native hadoop libraries is imperative + for the lzo and gzip compression codecs to work.

Usage

It is fairly simple to use the native hadoop libraries:

+ Take a look at the + supported platforms. +
+ Either download the pre-built + 32-bit i386-Linux native hadoop libraries (available as part of hadoop + distribution in lib/native directory) or + build them yourself. +
+ Ensure you have either or both of >zlib-1.2 and + >lzo2.0 packages for your platform installed; + depending on your needs. +

The bin/hadoop script ensures that the native hadoop + library is on the library path via the system property + -Djava.library.path=<path>.

To check everything went alright check the hadoop log files for:

+ + + DEBUG util.NativeCodeLoader - Trying to load the custom-built + native-hadoop library... + +
+ + + INFO util.NativeCodeLoader - Loaded the native-hadoop library + + +

If something goes wrong, then:

+ + + INFO util.NativeCodeLoader - Unable to load native-hadoop library for + your platform... using builtin-java classes where applicable + + +

Supported Platforms

Hadoop native library is supported only on *nix platforms only. + Unfortunately it is known not to work on Cygwin + and Mac OS X and has mainly been used on the + GNU/Linux platform.

It has been tested on the following GNU/Linux distributions:

+ +RHEL4/Fedora + +
+Ubuntu +
+Gentoo +

On all the above platforms a 32/64 bit Hadoop native library will work + with a respective 32/64 bit jvm.

Building Native Hadoop Libraries

Hadoop native library is written in + ANSI C and built using + the GNU autotools-chain (autoconf, autoheader, automake, autoscan, libtool). + This means it should be straight-forward to build them on any platform with + a standards compliant C compiler and the GNU autotools-chain. + See supported platforms.

In particular the various packages you would need on the target + platform are:

+ C compiler (e.g. GNU C Compiler) +
+ GNU Autools Chain: + autoconf, + automake, + libtool + +
+ zlib-development package (stable version >= 1.2.0) +
+ lzo-development package (stable version >= 2.0) +

Once you have the pre-requisites use the standard build.xml + and pass along the compile.native flag (set to + true) to build the native hadoop library:

+$ ant -Dcompile.native=true <target> +

The native hadoop library is not built by default since not everyone is + interested in building them.

You should see the newly-built native hadoop library in:

+$ build/native/<platform>/lib +

where <platform> is combination of the system-properties: + ${os.name}-${os.arch}-${sun.arch.data.model}; for e.g. + Linux-i386-32.

+ +

Notes

+ It is mandatory to have both the zlib and lzo + development packages on the target platform for building the + native hadoop library; however for deployment it is sufficient to + install zlib or lzo if you wish to use only one of them. +
+ It is necessary to have the correct 32/64 libraries of both zlib/lzo + depending on the 32/64 bit jvm for the target platform for + building/deployment of the native hadoop library. +

- +

+ +

Documentation

@@ -75,6 +120,9 @@ Map-Reduce Tutorial

+Native Hadoop Libraries +

Streaming

@@ -96,8 +144,17 @@

+ +

@@ -469,18 +526,27 @@

Modified: lucene/hadoop/trunk/docs/skin/CommonMessages_de.xml URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/docs/skin/CommonMessages_de.xml?rev=608973&r1=608972&r2=608973&view=diff ============================================================================== --- lucene/hadoop/trunk/docs/skin/CommonMessages_de.xml (original) +++ lucene/hadoop/trunk/docs/skin/CommonMessages_de.xml Fri Jan 4 11:52:39 2008 @@ -1,11 +1,11 @@

Apache > Lucene > Hadoop

- +

+ + Modified: lucene/hadoop/trunk/src/docs/src/documentation/content/xdocs/index.xml URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/docs/src/documentation/content/xdocs/index.xml?rev=608973&r1=608972&r2=608973&view=diff ============================================================================== --- lucene/hadoop/trunk/src/docs/src/documentation/content/xdocs/index.xml (original) +++ lucene/hadoop/trunk/src/docs/src/documentation/content/xdocs/index.xml Fri Jan 4 11:52:39 2008 @@ -19,6 +19,7 @@

Hadoop Cluster Setup

Hadoop Distributed File System

Hadoop Map-Reduce Tutorial

Native Hadoop Libraries

API Docs

Wiki

FAQ

Modified: lucene/hadoop/trunk/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml?rev=608973&r1=608972&r2=608973&view=diff ============================================================================== --- lucene/hadoop/trunk/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml (original) +++ lucene/hadoop/trunk/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml Fri Jan 4 11:52:39 2008 @@ -1109,8 +1109,14 @@ individual task.

- TextInputFormat is the default InputFormat. -

+ TextInputFormat is the default InputFormat.

+ +

If TextInputFormat is the InputFormat for a + given job, the framework detects input-files with the .gz and + .lzo extensions and automatically decompresses them using the + appropriate CompressionCodec. However, it must be noted that + compressed files with the above extensions cannot be split and + each compressed file is processed in its entirety by a single mapper.

InputSplit @@ -1336,6 +1342,70 @@ JobControl is a utility which encapsulates a set of Map-Reduce jobs and their dependencies.

+ +

+ Data Compression + +

Hadoop Map-Reduce provides facilities for the application-writer to + specify compression for both intermediate map-outputs and the + job-outputs i.e. output of the reduces. It also comes bundled with + + CompressionCodec implementations for the + zlib and lzo compression + algorithms. The gzip file format is also + supported.

+ +

Hadoop also provides native implementations of the above compression + codecs for reasons of both performance (zlib) and non-availability of + Java libraries (lzo). More details on their usage and availability are + available here.

+ +

+ Intermediate Outputs + +

Applications can control compression of intermediate map-outputs + via the + + JobConf.setCompressMapOutput(boolean) api and the + CompressionCodec to be used via the + + JobConf.setMapOutputCompressorClass(Class) api. Since + the intermediate map-outputs are always stored in the + SequenceFile + format, the + + SequenceFile.CompressionType (i.e. + + RECORD / + + BLOCK - defaults to RECORD) can be specified via the + + JobConf.setMapOutputCompressionType(SequenceFile.CompressionType) + api.

+ +

+ Job Outputs + +

Applications can control compression of job-outputs via the + + OutputFormatBase.setCompressOutput(JobConf, boolean) api and the + CompressionCodec to be used can be specified via the + + OutputFormatBase.setOutputCompressorClass(JobConf, Class) api.

+ +

If the job outputs are to be stored in the + + SequenceFileOutputFormat, the required + SequenceFile.CompressionType (i.e. RECORD / + BLOCK - defaults to RECORD)can be specified + via the + + SequenceFileOutputFormat.setOutputCompressionType(JobConf, + SequenceFile.CompressionType) api.

+ Added: lucene/hadoop/trunk/src/docs/src/documentation/content/xdocs/native_libraries.xml URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/docs/src/documentation/content/xdocs/native_libraries.xml?rev=608973&view=auto ============================================================================== --- lucene/hadoop/trunk/src/docs/src/documentation/content/xdocs/native_libraries.xml (added) +++ lucene/hadoop/trunk/src/docs/src/documentation/content/xdocs/native_libraries.xml Fri Jan 4 11:52:39 2008 @@ -0,0 +1,175 @@ + + + + + + +

+ Native Hadoop Libraries +

+ + + +

+ Purpose + +

+ +

+ Components + +

Hadoop currently has the following + + compression codecs as the native components:

zlib
gzip
lzo

+ +

Of the above, the availability of native hadoop libraries is imperative + for the lzo and gzip compression codecs to work.

+ +

+ Usage + +

It is fairly simple to use the native hadoop libraries:

+ +

+ Take a look at the + supported platforms. +
+ Either download the pre-built + 32-bit i386-Linux native hadoop libraries (available as part of hadoop + distribution in lib/native directory) or + build them yourself. +
+ Ensure you have either or both of >zlib-1.2 and + >lzo2.0 packages for your platform installed; + depending on your needs. +

+ +

The bin/hadoop script ensures that the native hadoop + library is on the library path via the system property + -Djava.library.path=<path>.

+ +

To check everything went alright check the hadoop log files for:

+ +

+ + DEBUG util.NativeCodeLoader - Trying to load the custom-built + native-hadoop library... +
+ + INFO util.NativeCodeLoader - Loaded the native-hadoop library + +

+ +

If something goes wrong, then:

+ + INFO util.NativeCodeLoader - Unable to load native-hadoop library for + your platform... using builtin-java classes where applicable + +

+ +

+ Supported Platforms + +

Hadoop native library is supported only on *nix platforms only. + Unfortunately it is known not to work on Cygwin + and Mac OS X and has mainly been used on the + GNU/Linux platform.

+ +

It has been tested on the following GNU/Linux distributions:

+ +

On all the above platforms a 32/64 bit Hadoop native library will work + with a respective 32/64 bit jvm.

+ +

+ Building Native Hadoop Libraries + +

+ +

In particular the various packages you would need on the target + platform are:

+ C compiler (e.g. GNU C Compiler) +
+ GNU Autools Chain: + autoconf, + automake, + libtool +
+ zlib-development package (stable version >= 1.2.0) +
+ lzo-development package (stable version >= 2.0) +

+ +

Once you have the pre-requisites use the standard build.xml + and pass along the compile.native flag (set to + true) to build the native hadoop library:

+ +

$ ant -Dcompile.native=true <target>

+ +

The native hadoop library is not built by default since not everyone is + interested in building them.

+ +

You should see the newly-built native hadoop library in:

+ +

$ build/native/<platform>/lib

+ +

where <platform> is combination of the system-properties: + ${os.name}-${os.arch}-${sun.arch.data.model}; for e.g. + Linux-i386-32.

+ +

+ Notes + +

+ It is mandatory to have both the zlib and lzo + development packages on the target platform for building the + native hadoop library; however for deployment it is sufficient to + install zlib or lzo if you wish to use only one of them. +
+ It is necessary to have the correct 32/64 libraries of both zlib/lzo + depending on the 32/64 bit jvm for the target platform for + building/deployment of the native hadoop library. +

+ + + Modified: lucene/hadoop/trunk/src/docs/src/documentation/content/xdocs/site.xml URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/docs/src/documentation/content/xdocs/site.xml?rev=608973&r1=608972&r2=608973&view=diff ============================================================================== --- lucene/hadoop/trunk/src/docs/src/documentation/content/xdocs/site.xml (original) +++ lucene/hadoop/trunk/src/docs/src/documentation/content/xdocs/site.xml Fri Jan 4 11:52:39 2008 @@ -22,6 +22,7 @@ + @@ -33,13 +34,20 @@ - + + +

+ + + + + @@ -63,6 +71,11 @@ + + + + + @@ -99,6 +112,9 @@ + + + @@ -113,6 +129,13 @@ + + + + + + +