hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: hadoop native libs 32 and 64 bit
Date Mon, 27 Aug 2012 17:16:52 GMT
Hi Steven,

You may also use the common-dev@ lists for development
discussions/issues around common elements :)

Just for some context, this was changed in 2.x by us via:
https://issues.apache.org/jira/browse/HADOOP-7874

On Mon, Aug 27, 2012 at 10:39 PM, Steven Willis <swillis@compete.com> wrote:
> Hi,
>
> I've been looking for both the 32 and 64 bit hadoop native libraries and it looks like
the existence and location of these libraries keeps changing between releases. I downloaded
the following releases:
>
> hadoop-0.22.0
> hadoop-0.23.0
> hadoop-0.23.1
> hadoop-1.0.1
> hadoop-1.0.2
> hadoop-1.0.3
> hadoop-2.0.0-alpha
> hadoop-2.0.1-alpha
>
> and looked for the location of 'libhadoop.so*' and ran 'file' on any results, this is
what I found:
>
> ./hadoop-0.22.0/lib/native/Linux-amd64-64/libhadoop.so:       ELF 64-bit LSB shared object
> ./hadoop-0.22.0/lib/native/Linux-amd64-64/libhadoop.so.1:     ELF 64-bit LSB shared object
> ./hadoop-0.22.0/lib/native/Linux-amd64-64/libhadoop.so.1.0.0: ELF 64-bit LSB shared object
> ./hadoop-0.23.0/lib/libhadoop.so:                             symbolic link to `libhadoop.so.1.0.0'
> ./hadoop-0.23.0/lib/libhadoop.so.1:                           symbolic link to `libhadoop.so.1.0.0'
> ./hadoop-0.23.0/lib/libhadoop.so.1.0.0:                       ELF 32-bit LSB shared object
> ./hadoop-0.23.1/lib/native/libhadoop.so:                      symbolic link to `libhadoop.so.1.0.0'
> ./hadoop-0.23.1/lib/native/libhadoop.so.1:                    symbolic link to `libhadoop.so.1.0.0'
> ./hadoop-0.23.1/lib/native/libhadoop.so.1.0.0:                ELF 32-bit LSB shared object
> ./hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so:        ELF 64-bit LSB shared object
> ./hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so.1:      ELF 64-bit LSB shared object
> ./hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so.1.0.0:  ELF 64-bit LSB shared object
> ./hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so:         ELF 32-bit LSB shared object
> ./hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so.1:       ELF 32-bit LSB shared object
> ./hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so.1.0.0:   ELF 32-bit LSB shared object
> ./hadoop-1.0.2/lib/native/Linux-amd64-64/libhadoop.so:        ELF 64-bit LSB shared object
> ./hadoop-1.0.2/lib/native/Linux-amd64-64/libhadoop.so.1:      ELF 64-bit LSB shared object
> ./hadoop-1.0.2/lib/native/Linux-amd64-64/libhadoop.so.1.0.0:  ELF 64-bit LSB shared object
> ./hadoop-1.0.2/lib/native/Linux-i386-32/libhadoop.so:         ELF 32-bit LSB shared object
> ./hadoop-1.0.2/lib/native/Linux-i386-32/libhadoop.so.1:       ELF 32-bit LSB shared object
> ./hadoop-1.0.2/lib/native/Linux-i386-32/libhadoop.so.1.0.0:   ELF 32-bit LSB shared object
> ./hadoop-1.0.3/lib/native/Linux-amd64-64/libhadoop.so:        ELF 64-bit LSB shared object
> ./hadoop-1.0.3/lib/native/Linux-amd64-64/libhadoop.so.1:      ELF 64-bit LSB shared object
> ./hadoop-1.0.3/lib/native/Linux-amd64-64/libhadoop.so.1.0.0:  ELF 64-bit LSB shared object
> ./hadoop-1.0.3/lib/native/Linux-i386-32/libhadoop.so:         ELF 32-bit LSB shared object
> ./hadoop-1.0.3/lib/native/Linux-i386-32/libhadoop.so.1:       ELF 32-bit LSB shared object
> ./hadoop-1.0.3/lib/native/Linux-i386-32/libhadoop.so.1.0.0:   ELF 32-bit LSB shared object
> ./hadoop-2.0.0-alpha/lib/native/libhadoop.so:                 symbolic link to `libhadoop.so.1.0.0'
> ./hadoop-2.0.0-alpha/lib/native/libhadoop.so.1:               symbolic link to `libhadoop.so.1.0.0'
> ./hadoop-2.0.0-alpha/lib/native/libhadoop.so.1.0.0:           ELF 32-bit LSB shared object
> ./hadoop-2.0.1-alpha/lib/native/libhadoop.so:                 symbolic link to `libhadoop.so.1.0.0'
> ./hadoop-2.0.1-alpha/lib/native/libhadoop.so.1:               symbolic link to `libhadoop.so.1.0.0'
> ./hadoop-2.0.1-alpha/lib/native/libhadoop.so.1.0.0:           ELF 32-bit LSB shared object
>
> It looks like in 0.22.0, hadoop was only released with 64 bit libs. They were put in
an arch directory under 'lib/native'. And there are copies for the .so files not symlinks.
Hadoop 0.23.0 has only 32 bit libs which are directly under 'lib' (no 'native' or arch dir),
and there are symlinks for the .so files. Hadoop 0.23.1 still has just 32 bit libs and symlinks,
but they're now under 'lib/native'. From 1.0.1 to 1.0.3 we have both 32 and 64 bit libs and
they're back under 'lib/native/ARCH', and we're back to having copies of the libs instead
of symlinks. In 2.0.0-alpha and 2.0.1-alpha they go back to 32 bit only and under 'lib/native',
and we get symlinks back.
>
> The following is a table of what we get with each release:
>
> release           32  64  archdir  symlinks
> hadoop-0.22.0      N   Y        Y         N
> hadoop-0.23.0      Y   N        N*        Y
> hadoop-0.23.1      Y   N        N         Y
> hadoop-1.0.1       Y   Y        Y         N
> hadoop-1.0.2       Y   Y        Y         N
> hadoop-1.0.3       Y   Y        Y         N
> hadoop-2.0.0-alpha Y   N        N         Y
> hadoop-2.0.1-alpha Y   N        N         Y
>
> *libs not even under 'native'
>
> Can the build process be changed to include 32 and 64 bit libs, put them under 'lib/native/ARCH',
and use symlinks for the .so an .so.1 files? Alternatively, maybe the native libraries should
be split out into separate downloads, so you'd have something like:
>
> hadoop-X.Y.Z.tar.gz
> hadoop-native-X.Y.Z.i386.tar.gz
> hadoop-native-X.Y.Z.x86_64.tar.gz
>
> Is this something I should open a JIRA for?
>
> -Steven Willis



-- 
Harsh J

Mime
View raw message