hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steven Willis <swil...@compete.com>
Subject RE: hadoop native libs 32 and 64 bit
Date Mon, 27 Aug 2012 18:11:03 GMT
(moving to common-dev)

Thanks Harsh,

So what's the final outcome of these changes? Do we get both 32 and 64 bit libraries in the
release tarball? Will they be underneath an arch dir, or directly under lib/native? I'm just
a bit confused because the issue you reference is:

"native libs should be under lib/native/ dir"
https://issues.apache.org/jira/browse/HADOOP-7874

but there's also this one:

"change location of the native libraries to lib instead of lib/native"
https://issues.apache.org/jira/browse/HADOOP-7996

-Steven Willis

> -----Original Message-----
> From: Harsh J [mailto:harsh@cloudera.com]
> Sent: Monday, August 27, 2012 1:17 PM
> To: user@hadoop.apache.org
> Subject: Re: hadoop native libs 32 and 64 bit
> 
> Hi Steven,
> 
> You may also use the common-dev@ lists for development
> discussions/issues around common elements :)
> 
> Just for some context, this was changed in 2.x by us via:
> https://issues.apache.org/jira/browse/HADOOP-7874
> 
> On Mon, Aug 27, 2012 at 10:39 PM, Steven Willis <swillis@compete.com>
> wrote:
> > Hi,
> >
> > I've been looking for both the 32 and 64 bit hadoop native libraries
> and it looks like the existence and location of these libraries keeps
> changing between releases. I downloaded the following releases:
> >
> > hadoop-0.22.0
> > hadoop-0.23.0
> > hadoop-0.23.1
> > hadoop-1.0.1
> > hadoop-1.0.2
> > hadoop-1.0.3
> > hadoop-2.0.0-alpha
> > hadoop-2.0.1-alpha
> >
> > and looked for the location of 'libhadoop.so*' and ran 'file' on any
> results, this is what I found:
> >
> > ./hadoop-0.22.0/lib/native/Linux-amd64-64/libhadoop.so:       ELF 64-
> bit LSB shared object
> > ./hadoop-0.22.0/lib/native/Linux-amd64-64/libhadoop.so.1:     ELF 64-
> bit LSB shared object
> > ./hadoop-0.22.0/lib/native/Linux-amd64-64/libhadoop.so.1.0.0: ELF 64-
> bit LSB shared object
> > ./hadoop-0.23.0/lib/libhadoop.so:
> symbolic link to `libhadoop.so.1.0.0'
> > ./hadoop-0.23.0/lib/libhadoop.so.1:
> symbolic link to `libhadoop.so.1.0.0'
> > ./hadoop-0.23.0/lib/libhadoop.so.1.0.0:                       ELF 32-
> bit LSB shared object
> > ./hadoop-0.23.1/lib/native/libhadoop.so:
> symbolic link to `libhadoop.so.1.0.0'
> > ./hadoop-0.23.1/lib/native/libhadoop.so.1:
> symbolic link to `libhadoop.so.1.0.0'
> > ./hadoop-0.23.1/lib/native/libhadoop.so.1.0.0:                ELF 32-
> bit LSB shared object
> > ./hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so:        ELF 64-
> bit LSB shared object
> > ./hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so.1:      ELF 64-
> bit LSB shared object
> > ./hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so.1.0.0:  ELF 64-
> bit LSB shared object
> > ./hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so:         ELF 32-
> bit LSB shared object
> > ./hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so.1:       ELF 32-
> bit LSB shared object
> > ./hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so.1.0.0:   ELF 32-
> bit LSB shared object
> > ./hadoop-1.0.2/lib/native/Linux-amd64-64/libhadoop.so:        ELF 64-
> bit LSB shared object
> > ./hadoop-1.0.2/lib/native/Linux-amd64-64/libhadoop.so.1:      ELF 64-
> bit LSB shared object
> > ./hadoop-1.0.2/lib/native/Linux-amd64-64/libhadoop.so.1.0.0:  ELF 64-
> bit LSB shared object
> > ./hadoop-1.0.2/lib/native/Linux-i386-32/libhadoop.so:         ELF 32-
> bit LSB shared object
> > ./hadoop-1.0.2/lib/native/Linux-i386-32/libhadoop.so.1:       ELF 32-
> bit LSB shared object
> > ./hadoop-1.0.2/lib/native/Linux-i386-32/libhadoop.so.1.0.0:   ELF 32-
> bit LSB shared object
> > ./hadoop-1.0.3/lib/native/Linux-amd64-64/libhadoop.so:        ELF 64-
> bit LSB shared object
> > ./hadoop-1.0.3/lib/native/Linux-amd64-64/libhadoop.so.1:      ELF 64-
> bit LSB shared object
> > ./hadoop-1.0.3/lib/native/Linux-amd64-64/libhadoop.so.1.0.0:  ELF 64-
> bit LSB shared object
> > ./hadoop-1.0.3/lib/native/Linux-i386-32/libhadoop.so:         ELF 32-
> bit LSB shared object
> > ./hadoop-1.0.3/lib/native/Linux-i386-32/libhadoop.so.1:       ELF 32-
> bit LSB shared object
> > ./hadoop-1.0.3/lib/native/Linux-i386-32/libhadoop.so.1.0.0:   ELF 32-
> bit LSB shared object
> > ./hadoop-2.0.0-alpha/lib/native/libhadoop.so:
> symbolic link to `libhadoop.so.1.0.0'
> > ./hadoop-2.0.0-alpha/lib/native/libhadoop.so.1:
> symbolic link to `libhadoop.so.1.0.0'
> > ./hadoop-2.0.0-alpha/lib/native/libhadoop.so.1.0.0:           ELF 32-
> bit LSB shared object
> > ./hadoop-2.0.1-alpha/lib/native/libhadoop.so:
> symbolic link to `libhadoop.so.1.0.0'
> > ./hadoop-2.0.1-alpha/lib/native/libhadoop.so.1:
> symbolic link to `libhadoop.so.1.0.0'
> > ./hadoop-2.0.1-alpha/lib/native/libhadoop.so.1.0.0:           ELF 32-
> bit LSB shared object
> >
> > It looks like in 0.22.0, hadoop was only released with 64 bit libs.
> They were put in an arch directory under 'lib/native'. And there are
> copies for the .so files not symlinks. Hadoop 0.23.0 has only 32 bit
> libs which are directly under 'lib' (no 'native' or arch dir), and
> there are symlinks for the .so files. Hadoop 0.23.1 still has just 32
> bit libs and symlinks, but they're now under 'lib/native'. From 1.0.1
> to 1.0.3 we have both 32 and 64 bit libs and they're back under
> 'lib/native/ARCH', and we're back to having copies of the libs instead
> of symlinks. In 2.0.0-alpha and 2.0.1-alpha they go back to 32 bit only
> and under 'lib/native', and we get symlinks back.
> >
> > The following is a table of what we get with each release:
> >
> > release           32  64  archdir  symlinks
> > hadoop-0.22.0      N   Y        Y         N
> > hadoop-0.23.0      Y   N        N*        Y
> > hadoop-0.23.1      Y   N        N         Y
> > hadoop-1.0.1       Y   Y        Y         N
> > hadoop-1.0.2       Y   Y        Y         N
> > hadoop-1.0.3       Y   Y        Y         N
> > hadoop-2.0.0-alpha Y   N        N         Y
> > hadoop-2.0.1-alpha Y   N        N         Y
> >
> > *libs not even under 'native'
> >
> > Can the build process be changed to include 32 and 64 bit libs, put
> them under 'lib/native/ARCH', and use symlinks for the .so an .so.1
> files? Alternatively, maybe the native libraries should be split out
> into separate downloads, so you'd have something like:
> >
> > hadoop-X.Y.Z.tar.gz
> > hadoop-native-X.Y.Z.i386.tar.gz
> > hadoop-native-X.Y.Z.x86_64.tar.gz
> >
> > Is this something I should open a JIRA for?
> >
> > -Steven Willis
> 
> 
> 
> --
> Harsh J

Mime
View raw message