hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Lucene-hadoop Wiki] Update of "NativeHadoop" by Arun C Murthy
Date Sat, 13 Jan 2007 11:44:07 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Lucene-hadoop Wiki" for change notification.

The following page has been changed by Arun C Murthy:
http://wiki.apache.org/lucene-hadoop/NativeHadoop

------------------------------------------------------------------------------
  Hadoop has native implementations of certain components for reasons of both performace &
non-availability of java-implementations. 
  This section describes the usage & details on how to build the native-libraries.
  
- === Native Hadoop Libraries ===
+ == Native Hadoop Libraries ==
  
+ Hadoop has the following native-libraries:
+    * [http://www.zlib.net zlib] compression codec - Reworked zlib codec using nio's direct-buffers
which gives us 60%-70% speedup. (more details [http://issues.apache.org/jira/browse/HADOOP-538
here]
+    * [http://www.oberhumer.com/opensource/lzo/ lzo] compression codec - Implemented due
to lack of java bindings for lzo . (more details [http://issues.apache.org/jira/browse/HADOOP-851
here])
+ 
+ == Usage ==
+ 
+ It is fairly simple to use the native-hadoop libraries:
+    * Take a look at the [#Platforms supported platforms]
+    * Either [http://www.apache.org/dyn/closer.cgi/lucene/hadoop/ download] the prebuilt
32-bit Linux native-hadoop libraries (available as part of hadoop distribution in lib/native)
or [#BuildNativeHadoop build] them yourself.
+    * Ensure you have either or/both of '''>zlib-1.2''' and '''>lzo2.0''' packages
for your platform installed; depending on your needs.
+ That's it!
+ 
+ The '''bin/hadoop''' script ensures that the native-hadoop library is on the ''library path''
via the system property ''-Djava.library.path='' (another alternative is to use the ''LD_LIBRARY_PATH''
variable, but the former is recommended).
+ 
+ To check everything went alright check the hadoop log files for:
+ [[BR]]
+ {{{DEBUG util.NativeCodeLoader - Trying to load the custom-built native-hadoop library...
+ INFO  util.NativeCodeLoader - Loaded the native-hadoop library}}}
+ 
+ God-forbid something goes wrong, then:
+ [[BR]]
+ {{{INFO util.NativeCodeLoader - Unable to load native-hadoop library for your platform...
using builtin-java classes where applicable
+ }}}
+ recheck the above steps.
+ 
+ [[Anchor(Platforms)]]
+ == Supported Platforms ==
+ 
+ Native-hadoop library is supported for '''*nix''' platforms only. Unfortunately it is known
not to work on [http://www.cygwin.com Cygwin] and [http://www.apple.com/macosx Mac OS X] and
has mainly been used on the Linux platform.
+ 
+ It has been tested on the following Linux distributions:
+    * [http://www.redhat.com/rhel/ RHEL4]/[http://fedora.redhat.com/ Fedora]
+    * [http://www.ubuntu.com Ubuntu]
+    * [http://www.gentoo.org Gentoo]
+ 

Mime
View raw message