Making use of Hadoop Native Libraries in HBase

+ Making use of Hadoop Native Libraries in HBase + The Hadoop shared library has a bunch of facility including + compression libraries and fast crc'ing. To make this facility available + to HBase, do the following. HBase/Hadoop will fall back to use + alternatives if it cannot find the native library versions -- or + fail outright if you asking for an explicit compressor and there is + no alternative available. + Lets presume your Hadoop shipped with a native library that + suits the platform you are running HBase on. To check if the Hadoop + native library is available to HBase, run the following tool (available in + Hadoop 2.1 and greater): + $ ./bin/hbase --config ~/conf_hbase org.apache.hadoop.util.NativeLibraryChecker +2014-08-26 13:15:38,717 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable +Native library checking: +hadoop: false +zlib: false +snappy: false +lz4: false +bzip2: false +2014-08-26 13:15:38,863 INFO [main] util.ExitUtil: Exiting with status 1 +Above shows that the native hadoop library is not available in HBase context. + + To fix the above, either copy the Hadoop native libraries local or symlink to + them if the Hadoop and HBase stalls are adjacent in the filesystem. + You could also point at their location by setting the LD_LIBRARY_PATH environment + variable. + Where the JVM looks to find native librarys is "system dependent" + (See java.lang.System#loadLibrary(name)). On linux, by default, + is going to look in lib/native/PLATFORM where PLATFORM + is the label for the platform your HBase is installed on. + On a local linux machine, it seems to be the concatenation of the java properties + os.name and os.arch followed by whether 32 or 64 bit. + HBase on startup prints out all of the java system properties so find the os.name and os.arch + in the log. For example: + .... + 2014-08-06 15:27:22,853 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux + 2014-08-06 15:27:22,853 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64 + ... + + So in this case, the PLATFORM string is Linux-amd64-64. + Copying the Hadoop native libraries or symlinking at lib/native/Linux-amd64-64 + will ensure they are found. Check with the Hadoop NativeLibraryChecker. + + + Here is example of how to point at the Hadoop libs with LD_LIBRARY_PATH + environment variable: + $ LD_LIBRARY_PATH=~/hadoop-2.5.0-SNAPSHOT/lib/native ./bin/hbase --config ~/conf_hbase org.apache.hadoop.util.NativeLibraryChecker +2014-08-26 13:42:49,332 INFO [main] bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native +2014-08-26 13:42:49,337 INFO [main] zlib.ZlibFactory: Successfully loaded & initialized native-zlib library +Native library checking: +hadoop: true /home/stack/hadoop-2.5.0-SNAPSHOT/lib/native/libhadoop.so.1.0.0 +zlib: true /lib64/libz.so.1 +snappy: true /usr/lib64/libsnappy.so.1 +lz4: true revision:99 +bzip2: true /lib64/libbz2.so.1 +Set in hbase-env.sh the LD_LIBRARY_PATH environment variable when starting your HBase. + +

Compressor Configuration, Installation, and Use @@ -5225,7 +5284,9 @@ This option should not normally be used, and it is not in -fixAll. Configure HBase For Compressors Before HBase can use a given compressor, its libraries need to be available. Due to licensing issues, only GZ compression is available to HBase (via native Java libraries) in - a default installation. + a default installation. Other compression libraries are available via the shared library + bundled with your hadoop. The hadoop native library needs to be findable when HBase + starts. See

Compressor Support On the Master A new configuration setting was introduced in HBase 0.95, to check the Master to