hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stephen mulcahy (Created) (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-7825) Hadoop wrapper script not picking up native libs correctly
Date Wed, 16 Nov 2011 15:38:53 GMT
Hadoop wrapper script not picking up native libs correctly
----------------------------------------------------------

                 Key: HADOOP-7825
                 URL: https://issues.apache.org/jira/browse/HADOOP-7825
             Project: Hadoop Common
          Issue Type: Bug
          Components: scripts
    Affects Versions: 0.20.205.0
         Environment: Debian 6.0 x64_64
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

            Reporter: stephen mulcahy


Originally discussed in https://mail-archives.apache.org/mod_mbox/hadoop-common-user/201111.mbox/%3C4EC3A3AE.7060402%40deri.org%3E

I'm testing out native lib support on our test amd64 test cluster 
running 0.20.205 running the following

./bin/hadoop jar hadoop-test-0.20.205.0.jar testsequencefile -seed 0 
-count 1000 -compressType RECORD xxx -codec 
org.apache.hadoop.io.compress.GzipCodec -check 2

it fails with

WARN util.NativeCodeLoader: Unable to load native-hadoop library for 
your platform... using builtin-java classes where applicable

Looking at

bin/hadoop

it seems to successfully detect that the native libs are available (they 
seem to come pre-compiled with 0.20.205 which is nice)

   if [ -d "${HADOOP_HOME}/lib/native" ]; then
     if [ "x$JAVA_LIBRARY_PATH" != "x" ]; then
 
JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}:${HADOOP_HOME}/lib/native/${JAVA_PLATFORM}
     else
       JAVA_LIBRARY_PATH=${HADOOP_HOME}/lib/native/${JAVA_PLATFORM}
     fi
   fi

and sets JAVA_LIBRARY_PATH to contain them.

Then in the following line, if ${HADOOP_HOME}/lib contains libhadoop.a 
(which is seems to in the stock tar) then it proceeds to ignore the 
native libs

   if [ -e "${HADOOP_PREFIX}/lib/libhadoop.a" ]; then
     JAVA_LIBRARY_PATH=${HADOOP_PREFIX}/lib
   fi

The libhadoop.a in ${HADOOP_HOME}/lib seems to be a copy of the lib/native/Linux-i386-32 going
from the filesizes (and also noted by https://mail-archives.apache.org/mod_mbox/hadoop-common-user/201111.mbox/%3CCAOcnVr2AzUDnN0LFhmTqUmAyujYtvhfkmMm_J0R-bmxw2wU+9A@mail.gmail.com%3E)

hadoop@testhbase01:~$ ls -la hadoop/lib/libhadoop.*
-rw-r--r-- 1 hadoop hadoop 237244 Oct  7 08:20 hadoop/lib/libhadoop.a
-rw-r--r-- 1 hadoop hadoop    877 Oct  7 08:20 hadoop/lib/libhadoop.la
-rw-r--r-- 1 hadoop hadoop 160438 Oct  7 08:20 hadoop/lib/libhadoop.so
-rw-r--r-- 1 hadoop hadoop 160438 Oct  7 08:19 hadoop/lib/libhadoop.so.1.0.0
hadoop@testhbase01:~$ ls -la hadoop/lib/native/Linux-i386-32/
total 728
drwxr-xr-x 3 hadoop hadoop   4096 Nov 15 14:05 .
drwxr-xr-x 5 hadoop hadoop   4096 Oct  7 08:24 ..
-rw-r--r-- 1 hadoop hadoop 237244 Oct  7 08:20 libhadoop.a
-rw-r--r-- 1 hadoop hadoop    877 Oct  7 08:20 libhadoop.la
-rw-r--r-- 1 hadoop hadoop 160438 Oct  7 08:20 libhadoop.so
-rw-r--r-- 1 hadoop hadoop 160438 Oct  7 08:20 libhadoop.so.1
-rw-r--r-- 1 hadoop hadoop 160438 Oct  7 08:20 libhadoop.so.1.0.0

A possible solution includes removing libhadoop.a and friends from ${HADOOP_HOME}/lib and
possibly also modifying the hadoop wrapper to remove 

   if [ -e "${HADOOP_PREFIX}/lib/libhadoop.a" ]; then
     JAVA_LIBRARY_PATH=${HADOOP_PREFIX}/lib
   fi

unless there is some other reason for this to exist.

This was also noted in a comment to HADOOP-6453

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message