hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steven Willis <swil...@compete.com>
Subject hadoop native libs 32 and 64 bit
Date Mon, 27 Aug 2012 17:09:24 GMT
Hi,

I've been looking for both the 32 and 64 bit hadoop native libraries and it looks like the
existence and location of these libraries keeps changing between releases. I downloaded the
following releases:

hadoop-0.22.0
hadoop-0.23.0
hadoop-0.23.1
hadoop-1.0.1
hadoop-1.0.2
hadoop-1.0.3
hadoop-2.0.0-alpha
hadoop-2.0.1-alpha

and looked for the location of 'libhadoop.so*' and ran 'file' on any results, this is what
I found:

./hadoop-0.22.0/lib/native/Linux-amd64-64/libhadoop.so:       ELF 64-bit LSB shared object
./hadoop-0.22.0/lib/native/Linux-amd64-64/libhadoop.so.1:     ELF 64-bit LSB shared object
./hadoop-0.22.0/lib/native/Linux-amd64-64/libhadoop.so.1.0.0: ELF 64-bit LSB shared object
./hadoop-0.23.0/lib/libhadoop.so:                             symbolic link to `libhadoop.so.1.0.0'
./hadoop-0.23.0/lib/libhadoop.so.1:                           symbolic link to `libhadoop.so.1.0.0'
./hadoop-0.23.0/lib/libhadoop.so.1.0.0:                       ELF 32-bit LSB shared object
./hadoop-0.23.1/lib/native/libhadoop.so:                      symbolic link to `libhadoop.so.1.0.0'
./hadoop-0.23.1/lib/native/libhadoop.so.1:                    symbolic link to `libhadoop.so.1.0.0'
./hadoop-0.23.1/lib/native/libhadoop.so.1.0.0:                ELF 32-bit LSB shared object
./hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so:        ELF 64-bit LSB shared object
./hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so.1:      ELF 64-bit LSB shared object
./hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so.1.0.0:  ELF 64-bit LSB shared object
./hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so:         ELF 32-bit LSB shared object
./hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so.1:       ELF 32-bit LSB shared object
./hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so.1.0.0:   ELF 32-bit LSB shared object
./hadoop-1.0.2/lib/native/Linux-amd64-64/libhadoop.so:        ELF 64-bit LSB shared object
./hadoop-1.0.2/lib/native/Linux-amd64-64/libhadoop.so.1:      ELF 64-bit LSB shared object
./hadoop-1.0.2/lib/native/Linux-amd64-64/libhadoop.so.1.0.0:  ELF 64-bit LSB shared object
./hadoop-1.0.2/lib/native/Linux-i386-32/libhadoop.so:         ELF 32-bit LSB shared object
./hadoop-1.0.2/lib/native/Linux-i386-32/libhadoop.so.1:       ELF 32-bit LSB shared object
./hadoop-1.0.2/lib/native/Linux-i386-32/libhadoop.so.1.0.0:   ELF 32-bit LSB shared object
./hadoop-1.0.3/lib/native/Linux-amd64-64/libhadoop.so:        ELF 64-bit LSB shared object
./hadoop-1.0.3/lib/native/Linux-amd64-64/libhadoop.so.1:      ELF 64-bit LSB shared object
./hadoop-1.0.3/lib/native/Linux-amd64-64/libhadoop.so.1.0.0:  ELF 64-bit LSB shared object
./hadoop-1.0.3/lib/native/Linux-i386-32/libhadoop.so:         ELF 32-bit LSB shared object
./hadoop-1.0.3/lib/native/Linux-i386-32/libhadoop.so.1:       ELF 32-bit LSB shared object
./hadoop-1.0.3/lib/native/Linux-i386-32/libhadoop.so.1.0.0:   ELF 32-bit LSB shared object
./hadoop-2.0.0-alpha/lib/native/libhadoop.so:                 symbolic link to `libhadoop.so.1.0.0'
./hadoop-2.0.0-alpha/lib/native/libhadoop.so.1:               symbolic link to `libhadoop.so.1.0.0'
./hadoop-2.0.0-alpha/lib/native/libhadoop.so.1.0.0:           ELF 32-bit LSB shared object
./hadoop-2.0.1-alpha/lib/native/libhadoop.so:                 symbolic link to `libhadoop.so.1.0.0'
./hadoop-2.0.1-alpha/lib/native/libhadoop.so.1:               symbolic link to `libhadoop.so.1.0.0'
./hadoop-2.0.1-alpha/lib/native/libhadoop.so.1.0.0:           ELF 32-bit LSB shared object

It looks like in 0.22.0, hadoop was only released with 64 bit libs. They were put in an arch
directory under 'lib/native'. And there are copies for the .so files not symlinks. Hadoop
0.23.0 has only 32 bit libs which are directly under 'lib' (no 'native' or arch dir), and
there are symlinks for the .so files. Hadoop 0.23.1 still has just 32 bit libs and symlinks,
but they're now under 'lib/native'. From 1.0.1 to 1.0.3 we have both 32 and 64 bit libs and
they're back under 'lib/native/ARCH', and we're back to having copies of the libs instead
of symlinks. In 2.0.0-alpha and 2.0.1-alpha they go back to 32 bit only and under 'lib/native',
and we get symlinks back.

The following is a table of what we get with each release:

release           32  64  archdir  symlinks
hadoop-0.22.0      N   Y        Y         N
hadoop-0.23.0      Y   N        N*        Y
hadoop-0.23.1      Y   N        N         Y
hadoop-1.0.1       Y   Y        Y         N
hadoop-1.0.2       Y   Y        Y         N
hadoop-1.0.3       Y   Y        Y         N
hadoop-2.0.0-alpha Y   N        N         Y
hadoop-2.0.1-alpha Y   N        N         Y

*libs not even under 'native'

Can the build process be changed to include 32 and 64 bit libs, put them under 'lib/native/ARCH',
and use symlinks for the .so an .so.1 files? Alternatively, maybe the native libraries should
be split out into separate downloads, so you'd have something like:

hadoop-X.Y.Z.tar.gz
hadoop-native-X.Y.Z.i386.tar.gz
hadoop-native-X.Y.Z.x86_64.tar.gz

Is this something I should open a JIRA for?

-Steven Willis

Mime
View raw message