hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Allen Wittenauer <awittena...@linkedin.com>
Subject Re: Native libraries on OS X
Date Mon, 22 Feb 2010 20:54:51 GMT
What happens if you use the test sequence file code:

bin/hadoop jar hadoop-0.20.1-test.jar testsequencefile -seed 0 -count 1000
-compressType RECORD blah.seq -codec org.apache.hadoop.io.compress.GzipCodec
-check



On 2/22/10 12:51 PM, "Derek Brown" <derek@media6degrees.com> wrote:

> I was (seemingly) able to build the native libraries, but still get the
> "unable to load native-hadoop library" message at run-time (with a simple
> app of my own to read/write a gzip-compressed sequence file).
> Based on https://issues.apache.org/jira/browse/HADOOP-3659, "Patch to allow
> hadoop native to compile on Mac OS X", and other research, I did the
> following:
> 
> $ cd hadoop-0.20.1/
> $ export
> LDFLAGS=-L/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Libraries
> $ patch -p0 -b < HADOOP-3659.patch
> $ cd src/native/
> $ # To regenerate the configure script:
> $ autoreconf -ifv
> $ cd ../..
> $ ant compile-native
> $ ls -l build/native/Mac_OS_X-x86_64-64/lib
> total 248
> -rw-r--r--  1 derek  staff  14007 Feb 22 13:26 Makefile
> lrwxr-xr-x  1 derek  staff     17 Feb 22 13:26 libhadoop.1.0.0.dylib@ ->
> libhadoop.1.dylib
> -rwxr-xr-x  1 derek  staff   8404 Feb 22 13:26 libhadoop.1.dylib*
> -rw-r--r--  1 derek  staff  84880 Feb 22 13:26 libhadoop.a
> lrwxr-xr-x  1 derek  staff     17 Feb 22 13:26 libhadoop.dylib@ ->
> libhadoop.1.dylib
> -rw-r--r--  1 derek  staff    900 Feb 22 13:26 libhadoop.la
> 
> But the this still comes up when running my code:
> 
> 10/02/22 15:10:29 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Exception in thread "main" java.lang.IllegalArgumentException: SequenceFile
> doesn't work with GzipCodec without native-hadoop code!
> 
> I'm running my code as follows:
> 
> $ hadoop jar my.jar com.media6.MyClass [args to MyClass here]
> 
> I'm on OS X 10.5.8 Leopard, with an Intel Core 2 Duo MacBook.
> 
> I didn't install any other libraries, which there are some references to in
> the Jira issue, since the build succeeded.
> 
> hadoop.native.lib is true by default, but to be explicit, I added it to
> core-site.xml anyway.
> 
> Thanks for any pointers,
> Derek


Mime
View raw message