Return-Path: 
 <common-user-return-20602-apmail-hadoop-common-user-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-user-archive@www.apache.org
Received: (qmail 87489 invoked from network); 22 Feb 2010 23:00:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Feb 2010 23:00:39 -0000
Received: (qmail 23352 invoked by uid 500); 22 Feb 2010 23:00:37 -0000
Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org
Received: (qmail 23286 invoked by uid 500); 22 Feb 2010 23:00:36 -0000
Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-user-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-user-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-user@hadoop.apache.org>
List-Id: <common-user.hadoop.apache.org>
Reply-To: common-user@hadoop.apache.org
Delivered-To: mailing list common-user@hadoop.apache.org
Received: (qmail 23276 invoked by uid 99); 22 Feb 2010 23:00:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Feb 2010 23:00:36 +0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of awittenauer@linkedin.com
 designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com)
 (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Feb 2010 23:00:28 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:User-Agent:Date:Subject:From:To:
   Message-ID:Thread-Topic:Thread-Index:In-Reply-To:
   Mime-version:Content-type:Content-transfer-encoding;
  b=g8E4Z8boIhBp1ygZzrPmdRDCQEUMBM+9mSNhaP7vb3TdOziTE+7DTBVg
   FTg1gCQu64CuwI2DoVlw0o7c1luCnzYb4zrLiEMs3GLxa1NeaSEfw3Tmf
   eIC9AJ91eAPd6S0;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1266879628; x=1298415628;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20Native=20libraries=20on=20OS=20X|Date:
   =20Mon,=2022=20Feb=202010=2015:00:04=20-0800|Message-ID:
   =20<C7A84C74.32F0%awittenauer@linkedin.com>|To:=20<common
   -user@hadoop.apache.org>|Mime-version:=201.0
   |Content-transfer-encoding:=207bit|In-Reply-To:=20<412c4e
   9f1002221448mfd25ffam1fbcaf68137305f0@mail.gmail.com>;
  bh=dkgxq7jQNlcqJfEKUmrVIsy31CbOm4xijf8uD6Iu6HM=;
  b=Leb02vI3pXRh3wD1GIaO9uJfM3q2/Yv41JvaAynIXwv9eH5Ji5HR5Srd
   TF8ruH97Hua/qyYWfjeh2N/DluFHk+c21cg7iSnVy+kkc+hESf5qgy+MD
   yl5Wy5gyIHP1ybm;
X-IronPort-AV: E=Sophos;i="4.49,521,1262592000";
   d="scan'208";a="11010193"
Received: from 172.16.19.141 ([172.16.19.141]) by CORP-MAIL.linkedin.biz
 ([172.18.46.135]) via Exchange Front-End Server mail-access.linkedin.biz
 ([172.18.46.133]) with Microsoft Exchange Server HTTP-DAV ;
 Mon, 22 Feb 2010 23:00:05 +0000
User-Agent: Microsoft-Entourage/12.10.0.080409
Date: Mon, 22 Feb 2010 15:00:04 -0800
Subject: Re: Native libraries on OS X
From: Allen Wittenauer <awittenauer@linkedin.com>
To: <common-user@hadoop.apache.org>
Message-ID: <C7A84C74.32F0%awittenauer@linkedin.com>
Thread-Topic: Native libraries on OS X
Thread-Index: Acq0EsTwLl7NUtDVIkqAJ+/fXFLKnw==
In-Reply-To: <412c4e9f1002221448mfd25ffam1fbcaf68137305f0@mail.gmail.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit


Did you compile this outside of ant?

If so, you probably got bit by the fact that the configure scripts assume
that the bitness level is shared via an environment variable.

[I really need to finish cleaning up my configure script patches. :( ]

On 2/22/10 2:48 PM, "Derek Brown" <derek@media6degrees.com> wrote:

> Aha- setting HADOOP_ROOT_LOOGER did indeed reveal more details:
> 
> 10/02/22 17:40:20 DEBUG util.NativeCodeLoader: Failed to load native-hadoop
> with error: java.lang.UnsatisfiedLinkError:
> /Users/derek/hadoop/hadoop-0.20.1/build/native/Mac_OS_X-x86_64-64/lib/libhadoo
> p.1.dylib:
> no suitable image found.  Did find:
> /Users/derek/hadoop/hadoop-0.20.1/build/native/Mac_OS_X-x86_64-64/lib/libhadoo
> p.1.dylib:
> mach-o, but wrong architecture
> 10/02/22 17:40:20 DEBUG util.NativeCodeLoader:
> java.library.path=/Users/derek/hadoop/hadoop-0.20.1/bin/../build/native/Mac_OS
> _X-x86_64-64/lib:/Users/derek/hadoop/hadoop-0.20.1/bin/../lib/native/Mac_OS_X-
> x86_64-64
> 
> So it looks like libhadoop.1.dylib is the wrong format.
> 
> $ file build/native/Mac_OS_X-x86_64-64/lib/libhadoop.1.dylib
> build/native/Mac_OS_X-x86_64-64/lib/libhadoop.1.dylib: Mach-O dynamically
> linked shared library i386
> 
> Thanks,
> Derek
> 
> 
>> From    Alex Kozlov <ale...@cloudera.com>
>> Subject    Re: Native libraries on OS X
>> Date    Mon, 22 Feb 2010 22:12:48 GMT
>> 
>> Sometimes another problem is shadowed by the file loading exception.  Could
>> you try to define HADOOP_ROOT_LOOGER=DEBUG,console rerun the command and
>> check the log?
>> 
>> On Mon, Feb 22, 2010 at 1:53 PM, Derek Brown <derek@media6degrees.com
>> wrote:
>> 
>>> I didn't set java.library.path explicitly because the bin/hadoop script
>>> sets
>>> it itself. I added a -x to the
>>> #!/usr/bin/env bash
>>> line, to have bash output the assignments and expansions it's doing, and
> I
>>> do see that it's setting it correctly and passing to the java executable
>>> call.
>>> 
>>> And same problem after setting mapred.child.java.opts.
>>> 
>>> Thanks,
>>> Derek
>>> 
>>> 
>>>> From    Alex Kozlov <ale...@cloudera.com>
>>>> Subject    Re: Native libraries on OS X
>>>> Date    Mon, 22 Feb 2010 21:03:41 GMT
>>> 
>>>> Have you set java.library.path?  Add "-*Djava.library.path=$HOME**/**
>>>> hadoop-0.20.1/build/native/Mac_OS_X-x86_64-64/lib"* to
>>>> mapred.child.java.opts.
>>>> 
>>>> On Mon, Feb 22, 2010 at 12:51 PM, Derek Brown <derek@media6degrees.com
>>>> wrote:
>>>> 
>>>>> I was (seemingly) able to build the native libraries, but still get
> the
>>>>> "unable to load native-hadoop library" message at run-time (with a
>>> simple
>>>>> app of my own to read/write a gzip-compressed sequence file).
>>>>> Based on https://issues.apache.org/jira/browse/HADOOP-3659, "Patch to
>>>>> allow
>>>>> hadoop native to compile on Mac OS X", and other research, I did the
>>>>> following:
>>>>> 
>>>>> $ cd hadoop-0.20.1/
>>>>> $ export
>>>>> 
>>>>> 
>>> 
>>> 
> LDFLAGS=-L/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Libraries
>>>>> $ patch -p0 -b < HADOOP-3659.patch
>>>>> $ cd src/native/
>>>>> $ # To regenerate the configure script:
>>>>> $ autoreconf -ifv
>>>>> $ cd ../..
>>>>> $ ant compile-native
>>>>> $ ls -l build/native/Mac_OS_X-x86_64-64/lib
>>>>> total 248
>>>>> -rw-r--r--  1 derek  staff  14007 Feb 22 13:26 Makefile
>>>>> lrwxr-xr-x  1 derek  staff     17 Feb 22 13:26 libhadoop.1.0.0.dylib@
> ->
>>>>> libhadoop.1.dylib
>>>>> -rwxr-xr-x  1 derek  staff   8404 Feb 22 13:26 libhadoop.1.dylib*
>>>>> -rw-r--r--  1 derek  staff  84880 Feb 22 13:26 libhadoop.a
>>>>> lrwxr-xr-x  1 derek  staff     17 Feb 22 13:26 libhadoop.dylib@ ->
>>>>> libhadoop.1.dylib
>>>>> -rw-r--r--  1 derek  staff    900 Feb 22 13:26 libhadoop.la
>>>>> 
>>>>> But the this still comes up when running my code:
>>>>> 
>>>>> 10/02/22 15:10:29 WARN util.NativeCodeLoader: Unable to load
>>> native-hadoop
>>>>> library for your platform... using builtin-java classes where
> applicable
>>>>> Exception in thread "main" java.lang.IllegalArgumentException:
>>> SequenceFile
>>>>> doesn't work with GzipCodec without native-hadoop code!
>>>>> 
>>>>> I'm running my code as follows:
>>>>> 
>>>>> $ hadoop jar my.jar com.media6.MyClass [args to MyClass here]
>>>>> 
>>>>> I'm on OS X 10.5.8 Leopard, with an Intel Core 2 Duo MacBook.
>>>>> 
>>>>> I didn't install any other libraries, which there are some references
> to
>>> in
>>>>> the Jira issue, since the build succeeded.
>>>>> 
>>>>> hadoop.native.lib is true by default, but to be explicit, I added it
> to
>>>>> core-site.xml anyway.
>>>>> 
>>>>> Thanks for any pointers,
>>>>> Derek