hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Allen Wittenauer <awittena...@linkedin.com>
Subject Re: Native libraries on OS X
Date Mon, 22 Feb 2010 23:00:04 GMT

Did you compile this outside of ant?

If so, you probably got bit by the fact that the configure scripts assume
that the bitness level is shared via an environment variable.

[I really need to finish cleaning up my configure script patches. :( ]

On 2/22/10 2:48 PM, "Derek Brown" <derek@media6degrees.com> wrote:

> Aha- setting HADOOP_ROOT_LOOGER did indeed reveal more details:
> 
> 10/02/22 17:40:20 DEBUG util.NativeCodeLoader: Failed to load native-hadoop
> with error: java.lang.UnsatisfiedLinkError:
> /Users/derek/hadoop/hadoop-0.20.1/build/native/Mac_OS_X-x86_64-64/lib/libhadoo
> p.1.dylib:
> no suitable image found.  Did find:
> /Users/derek/hadoop/hadoop-0.20.1/build/native/Mac_OS_X-x86_64-64/lib/libhadoo
> p.1.dylib:
> mach-o, but wrong architecture
> 10/02/22 17:40:20 DEBUG util.NativeCodeLoader:
> java.library.path=/Users/derek/hadoop/hadoop-0.20.1/bin/../build/native/Mac_OS
> _X-x86_64-64/lib:/Users/derek/hadoop/hadoop-0.20.1/bin/../lib/native/Mac_OS_X-
> x86_64-64
> 
> So it looks like libhadoop.1.dylib is the wrong format.
> 
> $ file build/native/Mac_OS_X-x86_64-64/lib/libhadoop.1.dylib
> build/native/Mac_OS_X-x86_64-64/lib/libhadoop.1.dylib: Mach-O dynamically
> linked shared library i386
> 
> Thanks,
> Derek
> 
> 
>> From    Alex Kozlov <ale...@cloudera.com>
>> Subject    Re: Native libraries on OS X
>> Date    Mon, 22 Feb 2010 22:12:48 GMT
>> 
>> Sometimes another problem is shadowed by the file loading exception.  Could
>> you try to define HADOOP_ROOT_LOOGER=DEBUG,console rerun the command and
>> check the log?
>> 
>> On Mon, Feb 22, 2010 at 1:53 PM, Derek Brown <derek@media6degrees.com
>> wrote:
>> 
>>> I didn't set java.library.path explicitly because the bin/hadoop script
>>> sets
>>> it itself. I added a -x to the
>>> #!/usr/bin/env bash
>>> line, to have bash output the assignments and expansions it's doing, and
> I
>>> do see that it's setting it correctly and passing to the java executable
>>> call.
>>> 
>>> And same problem after setting mapred.child.java.opts.
>>> 
>>> Thanks,
>>> Derek
>>> 
>>> 
>>>> From    Alex Kozlov <ale...@cloudera.com>
>>>> Subject    Re: Native libraries on OS X
>>>> Date    Mon, 22 Feb 2010 21:03:41 GMT
>>> 
>>>> Have you set java.library.path?  Add "-*Djava.library.path=$HOME**/**
>>>> hadoop-0.20.1/build/native/Mac_OS_X-x86_64-64/lib"* to
>>>> mapred.child.java.opts.
>>>> 
>>>> On Mon, Feb 22, 2010 at 12:51 PM, Derek Brown <derek@media6degrees.com
>>>> wrote:
>>>> 
>>>>> I was (seemingly) able to build the native libraries, but still get
> the
>>>>> "unable to load native-hadoop library" message at run-time (with a
>>> simple
>>>>> app of my own to read/write a gzip-compressed sequence file).
>>>>> Based on https://issues.apache.org/jira/browse/HADOOP-3659, "Patch to
>>>>> allow
>>>>> hadoop native to compile on Mac OS X", and other research, I did the
>>>>> following:
>>>>> 
>>>>> $ cd hadoop-0.20.1/
>>>>> $ export
>>>>> 
>>>>> 
>>> 
>>> 
> LDFLAGS=-L/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Libraries
>>>>> $ patch -p0 -b < HADOOP-3659.patch
>>>>> $ cd src/native/
>>>>> $ # To regenerate the configure script:
>>>>> $ autoreconf -ifv
>>>>> $ cd ../..
>>>>> $ ant compile-native
>>>>> $ ls -l build/native/Mac_OS_X-x86_64-64/lib
>>>>> total 248
>>>>> -rw-r--r--  1 derek  staff  14007 Feb 22 13:26 Makefile
>>>>> lrwxr-xr-x  1 derek  staff     17 Feb 22 13:26 libhadoop.1.0.0.dylib@
> ->
>>>>> libhadoop.1.dylib
>>>>> -rwxr-xr-x  1 derek  staff   8404 Feb 22 13:26 libhadoop.1.dylib*
>>>>> -rw-r--r--  1 derek  staff  84880 Feb 22 13:26 libhadoop.a
>>>>> lrwxr-xr-x  1 derek  staff     17 Feb 22 13:26 libhadoop.dylib@ ->
>>>>> libhadoop.1.dylib
>>>>> -rw-r--r--  1 derek  staff    900 Feb 22 13:26 libhadoop.la
>>>>> 
>>>>> But the this still comes up when running my code:
>>>>> 
>>>>> 10/02/22 15:10:29 WARN util.NativeCodeLoader: Unable to load
>>> native-hadoop
>>>>> library for your platform... using builtin-java classes where
> applicable
>>>>> Exception in thread "main" java.lang.IllegalArgumentException:
>>> SequenceFile
>>>>> doesn't work with GzipCodec without native-hadoop code!
>>>>> 
>>>>> I'm running my code as follows:
>>>>> 
>>>>> $ hadoop jar my.jar com.media6.MyClass [args to MyClass here]
>>>>> 
>>>>> I'm on OS X 10.5.8 Leopard, with an Intel Core 2 Duo MacBook.
>>>>> 
>>>>> I didn't install any other libraries, which there are some references
> to
>>> in
>>>>> the Jira issue, since the build succeeded.
>>>>> 
>>>>> hadoop.native.lib is true by default, but to be explicit, I added it
> to
>>>>> core-site.xml anyway.
>>>>> 
>>>>> Thanks for any pointers,
>>>>> Derek


Mime
View raw message