hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Harsh J (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-7825) Hadoop wrapper script not picking up native libs correctly
Date Thu, 17 Nov 2011 05:26:54 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151822#comment-13151822
] 

Harsh J commented on HADOOP-7825:
---------------------------------

Two things I'll need to investigate here before proposing to remove such a check:

- What are native libs doing in lib/, when you can't guarantee their platform/arch without
using external tools?
- What utilizes their presence in lib/ in 205 today.

Clearly, using 'file', tells you that those under lib/* are 32-bit binaries in the tar we
ship today. This breaks native code loading in all 64-bit deployments.
                
> Hadoop wrapper script not picking up native libs correctly
> ----------------------------------------------------------
>
>                 Key: HADOOP-7825
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7825
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 0.20.205.0
>         Environment: Debian 6.0 x64_64
> java version "1.6.0_26"
> Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)
>            Reporter: stephen mulcahy
>
> Originally discussed in https://mail-archives.apache.org/mod_mbox/hadoop-common-user/201111.mbox/%3C4EC3A3AE.7060402%40deri.org%3E
> I'm testing out native lib support on our test amd64 test cluster 
> running 0.20.205 running the following
> ./bin/hadoop jar hadoop-test-0.20.205.0.jar testsequencefile -seed 0 
> -count 1000 -compressType RECORD xxx -codec 
> org.apache.hadoop.io.compress.GzipCodec -check 2
> it fails with
> WARN util.NativeCodeLoader: Unable to load native-hadoop library for 
> your platform... using builtin-java classes where applicable
> Looking at
> bin/hadoop
> it seems to successfully detect that the native libs are available (they 
> seem to come pre-compiled with 0.20.205 which is nice)
>    if [ -d "${HADOOP_HOME}/lib/native" ]; then
>      if [ "x$JAVA_LIBRARY_PATH" != "x" ]; then
>  
> JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}:${HADOOP_HOME}/lib/native/${JAVA_PLATFORM}
>      else
>        JAVA_LIBRARY_PATH=${HADOOP_HOME}/lib/native/${JAVA_PLATFORM}
>      fi
>    fi
> and sets JAVA_LIBRARY_PATH to contain them.
> Then in the following line, if ${HADOOP_HOME}/lib contains libhadoop.a 
> (which is seems to in the stock tar) then it proceeds to ignore the 
> native libs
>    if [ -e "${HADOOP_PREFIX}/lib/libhadoop.a" ]; then
>      JAVA_LIBRARY_PATH=${HADOOP_PREFIX}/lib
>    fi
> The libhadoop.a in ${HADOOP_HOME}/lib seems to be a copy of the lib/native/Linux-i386-32
going from the filesizes (and also noted by https://mail-archives.apache.org/mod_mbox/hadoop-common-user/201111.mbox/%3CCAOcnVr2AzUDnN0LFhmTqUmAyujYtvhfkmMm_J0R-bmxw2wU+9A@mail.gmail.com%3E)
> hadoop@testhbase01:~$ ls -la hadoop/lib/libhadoop.*
> -rw-r--r-- 1 hadoop hadoop 237244 Oct  7 08:20 hadoop/lib/libhadoop.a
> -rw-r--r-- 1 hadoop hadoop    877 Oct  7 08:20 hadoop/lib/libhadoop.la
> -rw-r--r-- 1 hadoop hadoop 160438 Oct  7 08:20 hadoop/lib/libhadoop.so
> -rw-r--r-- 1 hadoop hadoop 160438 Oct  7 08:19 hadoop/lib/libhadoop.so.1.0.0
> hadoop@testhbase01:~$ ls -la hadoop/lib/native/Linux-i386-32/
> total 728
> drwxr-xr-x 3 hadoop hadoop   4096 Nov 15 14:05 .
> drwxr-xr-x 5 hadoop hadoop   4096 Oct  7 08:24 ..
> -rw-r--r-- 1 hadoop hadoop 237244 Oct  7 08:20 libhadoop.a
> -rw-r--r-- 1 hadoop hadoop    877 Oct  7 08:20 libhadoop.la
> -rw-r--r-- 1 hadoop hadoop 160438 Oct  7 08:20 libhadoop.so
> -rw-r--r-- 1 hadoop hadoop 160438 Oct  7 08:20 libhadoop.so.1
> -rw-r--r-- 1 hadoop hadoop 160438 Oct  7 08:20 libhadoop.so.1.0.0
> A possible solution includes removing libhadoop.a and friends from ${HADOOP_HOME}/lib
and possibly also modifying the hadoop wrapper to remove 
>    if [ -e "${HADOOP_PREFIX}/lib/libhadoop.a" ]; then
>      JAVA_LIBRARY_PATH=${HADOOP_PREFIX}/lib
>    fi
> unless there is some other reason for this to exist.
> This was also noted in a comment to HADOOP-6453

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message