Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of swillis@compete.com designates
 63.118.96.206 as permitted sender)
From: Steven Willis <swillis@compete.com>
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Date: Mon, 27 Aug 2012 13:09:24 -0400
Subject: hadoop native libs 32 and 64 bit
Thread-Topic: hadoop native libs 32 and 64 bit
Thread-Index: Ac2EdrUED65uOl/KQHiaGAHYUuoSUQ==
Message-ID: 
 <19AD42E3F64F0F468A305399D0DF39D92EA4521523@winops07.win.compete.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hi,

I've been looking for both the 32 and 64 bit hadoop native libraries and it=
 looks like the existence and location of these libraries keeps changing be=
tween releases. I downloaded the following releases:

hadoop-0.22.0
hadoop-0.23.0
hadoop-0.23.1
hadoop-1.0.1
hadoop-1.0.2
hadoop-1.0.3
hadoop-2.0.0-alpha
hadoop-2.0.1-alpha

and looked for the location of 'libhadoop.so*' and ran 'file' on any result=
s, this is what I found:

./hadoop-0.22.0/lib/native/Linux-amd64-64/libhadoop.so:       ELF 64-bit LS=
B shared object
./hadoop-0.22.0/lib/native/Linux-amd64-64/libhadoop.so.1:     ELF 64-bit LS=
B shared object
./hadoop-0.22.0/lib/native/Linux-amd64-64/libhadoop.so.1.0.0: ELF 64-bit LS=
B shared object
./hadoop-0.23.0/lib/libhadoop.so:                             symbolic link=
 to `libhadoop.so.1.0.0'
./hadoop-0.23.0/lib/libhadoop.so.1:                           symbolic link=
 to `libhadoop.so.1.0.0'
./hadoop-0.23.0/lib/libhadoop.so.1.0.0:                       ELF 32-bit LS=
B shared object
./hadoop-0.23.1/lib/native/libhadoop.so:                      symbolic link=
 to `libhadoop.so.1.0.0'
./hadoop-0.23.1/lib/native/libhadoop.so.1:                    symbolic link=
 to `libhadoop.so.1.0.0'
./hadoop-0.23.1/lib/native/libhadoop.so.1.0.0:                ELF 32-bit LS=
B shared object
./hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so:        ELF 64-bit LS=
B shared object
./hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so.1:      ELF 64-bit LS=
B shared object
./hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so.1.0.0:  ELF 64-bit LS=
B shared object
./hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so:         ELF 32-bit LS=
B shared object
./hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so.1:       ELF 32-bit LS=
B shared object
./hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so.1.0.0:   ELF 32-bit LS=
B shared object
./hadoop-1.0.2/lib/native/Linux-amd64-64/libhadoop.so:        ELF 64-bit LS=
B shared object
./hadoop-1.0.2/lib/native/Linux-amd64-64/libhadoop.so.1:      ELF 64-bit LS=
B shared object
./hadoop-1.0.2/lib/native/Linux-amd64-64/libhadoop.so.1.0.0:  ELF 64-bit LS=
B shared object
./hadoop-1.0.2/lib/native/Linux-i386-32/libhadoop.so:         ELF 32-bit LS=
B shared object
./hadoop-1.0.2/lib/native/Linux-i386-32/libhadoop.so.1:       ELF 32-bit LS=
B shared object
./hadoop-1.0.2/lib/native/Linux-i386-32/libhadoop.so.1.0.0:   ELF 32-bit LS=
B shared object
./hadoop-1.0.3/lib/native/Linux-amd64-64/libhadoop.so:        ELF 64-bit LS=
B shared object
./hadoop-1.0.3/lib/native/Linux-amd64-64/libhadoop.so.1:      ELF 64-bit LS=
B shared object
./hadoop-1.0.3/lib/native/Linux-amd64-64/libhadoop.so.1.0.0:  ELF 64-bit LS=
B shared object
./hadoop-1.0.3/lib/native/Linux-i386-32/libhadoop.so:         ELF 32-bit LS=
B shared object
./hadoop-1.0.3/lib/native/Linux-i386-32/libhadoop.so.1:       ELF 32-bit LS=
B shared object
./hadoop-1.0.3/lib/native/Linux-i386-32/libhadoop.so.1.0.0:   ELF 32-bit LS=
B shared object
./hadoop-2.0.0-alpha/lib/native/libhadoop.so:                 symbolic link=
 to `libhadoop.so.1.0.0'
./hadoop-2.0.0-alpha/lib/native/libhadoop.so.1:               symbolic link=
 to `libhadoop.so.1.0.0'
./hadoop-2.0.0-alpha/lib/native/libhadoop.so.1.0.0:           ELF 32-bit LS=
B shared object
./hadoop-2.0.1-alpha/lib/native/libhadoop.so:                 symbolic link=
 to `libhadoop.so.1.0.0'
./hadoop-2.0.1-alpha/lib/native/libhadoop.so.1:               symbolic link=
 to `libhadoop.so.1.0.0'
./hadoop-2.0.1-alpha/lib/native/libhadoop.so.1.0.0:           ELF 32-bit LS=
B shared object

It looks like in 0.22.0, hadoop was only released with 64 bit libs. They we=
re put in an arch directory under 'lib/native'. And there are copies for th=
e .so files not symlinks. Hadoop 0.23.0 has only 32 bit libs which are dire=
ctly under 'lib' (no 'native' or arch dir), and there are symlinks for the =
.so files. Hadoop 0.23.1 still has just 32 bit libs and symlinks, but they'=
re now under 'lib/native'. From 1.0.1 to 1.0.3 we have both 32 and 64 bit l=
ibs and they're back under 'lib/native/ARCH', and we're back to having copi=
es of the libs instead of symlinks. In 2.0.0-alpha and 2.0.1-alpha they go =
back to 32 bit only and under 'lib/native', and we get symlinks back.

The following is a table of what we get with each release:

release           32  64  archdir  symlinks
hadoop-0.22.0      N   Y        Y         N
hadoop-0.23.0      Y   N        N*        Y
hadoop-0.23.1      Y   N        N         Y
hadoop-1.0.1       Y   Y        Y         N
hadoop-1.0.2       Y   Y        Y         N
hadoop-1.0.3       Y   Y        Y         N
hadoop-2.0.0-alpha Y   N        N         Y
hadoop-2.0.1-alpha Y   N        N         Y

*libs not even under 'native'

Can the build process be changed to include 32 and 64 bit libs, put them un=
der 'lib/native/ARCH', and use symlinks for the .so an .so.1 files? Alterna=
tively, maybe the native libraries should be split out into separate downlo=
ads, so you'd have something like:

hadoop-X.Y.Z.tar.gz
hadoop-native-X.Y.Z.i386.tar.gz
hadoop-native-X.Y.Z.x86_64.tar.gz

Is this something I should open a JIRA for?

-Steven Willis