Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-dev@hadoop.apache.org
Received-SPF: pass (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=gs97jJapyTzKczZ8BRW8u8sKwbN4odLh9a696hlCEXl37a/zzva05DL0sFbXXq1NNWfh/5w7rJbr0piglxHIq1bD3e5e7FgHo2fdb55RI+YTpesZan3KFAuBXzRFg8yZf/ggtf5A+v1+3/bubDolnBcVj0MwxroHKnonbeYStHU=;
Message-ID: <943627.34385.qm@web110701.mail.gq1.yahoo.com>
References: <BANLkTi=0Duq1YO48Jj+bxMQ1F_Qo3Tt_Fg@mail.gmail.com>
 <BANLkTikeEJ7Jrh1u9CJTtkvNJTqQzff37g@mail.gmail.com>
 <BANLkTimfayA9uZ=8Zquq7UZtwGrnqion+g@mail.gmail.com>
 <BANLkTi=-65AaSZ0eWNFm211G+xKXT1e-MQ@mail.gmail.com>
Date: Mon, 23 May 2011 08:46:45 -0700 (PDT)
From: Bharath Mundlapudi <bharathwork@yahoo.com>
Reply-To: Bharath Mundlapudi <bharathwork@yahoo.com>
Subject: Re: Hadoop native builds fail on ARM due to -m32
To: "common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>
In-Reply-To: <BANLkTi=-65AaSZ0eWNFm211G+xKXT1e-MQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-177622805-1306165605=:34385"

--0-177622805-1306165605=:34385
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Adding ARM processor support to Hadoop is great. Reducing power consumption=
 on Hadoop grids is a plus.=0A=0A=0AHi Trevor,=0A=0AYou have mentioned that=
 - "other tests/benchmarks have run much faster". This information is good =
to know. Can you please tell us which areas you are seeing improvements on =
w.r.t ARM compared to others. Is this public information? =0A=0A=0A-Bharath=
=0A=0A=0A=0A________________________________=0AFrom: Eli Collins <eli@cloud=
era.com>=0ATo: common-dev@hadoop.apache.org=0ASent: Sunday, May 22, 2011 8:=
38 PM=0ASubject: Re: Hadoop native builds fail on ARM due to -m32=0A=0AHey =
Trevor,=0A=0AThanks for all the info.=A0 I took a quick look at HADOOP-7276=
 and=0AHDFS-1920, haven't gotten a chance for a full review yet but they=0A=
don't look like they'll be a burden, and if they get Hadoop running on=0AAR=
M that's great!=0A=0AThanks,=0AEli=0A=0AOn Fri, May 20, 2011 at 4:27 PM, Tr=
evor Robinson <trevor@scurrilous.com> wrote:=0A> Hi Eli,=0A>=0A> On Thu, Ma=
y 19, 2011 at 1:39 PM, Eli Collins <eli@cloudera.com> wrote:=0A>> Thanks fo=
r contributing. =A0 Supporting ARM on Hadoop will require a=0A>> number of =
different changes right? Eg given that Hadoop currently=0A>> depends on som=
e Sun-specific classes, and requires a Sun-compatible=0A>> JVM you'll have =
to work around this dependency somehow, there's not a=0A>> Sun JVM for ARM =
right?=0A>=0A> Actually, there is a Sun JVM for ARM, and it works quite wel=
l:=0A>=0A> http://www.oracle.com/technetwork/java/embedded/downloads/index.=
html=0A>=0A> Currently, it's just a JRE, so you have to use another JDK for=
 javac,=0A> etc., but I'm optimistic that we'll see a Sun Java SE JDK for A=
RM=0A> servers one of these days, given all the ARM server activity from=0A=
> Calxeda [http://www.theregister.co.uk/2011/03/14/calxeda_arm_server/],=0A=
> Marvell, and nVidia=0A> [http://www.channelregister.co.uk/2011/01/05/nvid=
ia_arm_pc_server_chip/].=0A>=0A> With the patches I submitted, Hadoop build=
s completely and nearly all=0A> of the Commons and HDFS unit tests pass wit=
h OpenJDK on ARM. (Some of=0A> the Map/Reduce unit tests have some crashes =
due to a bug in the=0A> OpenJDK build I'm using.) I need to re-run the unit=
 tests with the Sun=0A> JRE and see if they pass; other tests/benchmarks ha=
ve run much faster=0A> and more reliably with the Sun JRE, so I anticipate =
better results.=0A> I've run tests like TestDFSIO with the Sun JRE and have=
 had no=0A> problems.=0A>=0A>> If there's a handful of additional changes t=
hen let's make an umbrella=0A>> jira for Hadoop ARM support and make the is=
sues you've already filed=0A>> sub-tasks. You can ping me off-line on how t=
o that if you want.=0A>> Supporting non-x86 processors and non-gcc compiler=
s is an additional=0A>> maintenance burden on the project so it would be he=
lpful to have an=0A>> end-game figured out so these patches don't bitrot in=
 the meantime.=0A>=0A> I really don't anticipate any additional changes at =
this point. No=0A> Java or C++ code changes have been necessary; it's simpl=
y removing=0A> -m32 from CFLAGS/LDFLAGS and adding ARM to the list of proce=
ssors in=0A> apsupport.m4 (which contains lots of other unsupported process=
ors=0A> anyway). And just to be clear, pretty much everyone uses gcc for=0A=
> compilation on ARM, so supporting another compiler is unnecessary for=0A>=
 this.=0A>=0A> I certainly don't want to increase maintenance burden at thi=
s point,=0A> especially given that data center-grade ARM servers are still =
in the=0A> prototype stage. OTOH, these changes seem pretty trivial to me, =
and=0A> allow other developers (particularly those evaluating ARM and those=
=0A> involved in the Ubuntu ARM Server 11.10 release this fall:=0A> https:/=
/blueprints.launchpad.net/ubuntu/+spec/server-o-arm-server) to=0A> get Hado=
op up and running without having to patch the build.=0A>=0A> I'll follow up=
 offline though, so I can better understand any concerns=0A> you may still =
have.=0A>=0A> Thanks,=0A> Trevor=0A>=0A>> On Tue, May 10, 2011 at 5:13 PM, =
Trevor Robinson <trevor@scurrilous.com> wrote:=0A>>> Is the native build fa=
iling on ARM (where gcc doesn't support -m32) a=0A>>> known issue, and is t=
here a workaround or fix pending?=0A>>>=0A>>> $ ant -Dcompile.native=3Dtrue=
=0A>>> ...=0A>>> =A0 =A0 =A0[exec] make =A0all-am=0A>>> =A0 =A0 =A0[exec] m=
ake[1]: Entering directory=0A>>> `/home/trobinson/dev/hadoop-common/build/n=
ative/Linux-arm-32'=0A>>> =A0 =A0 =A0[exec] /bin/bash ./libtool =A0--tag=3D=
CC =A0 --mode=3Dcompile gcc=0A>>> -DHAVE_CONFIG_H -I. -I/home/trobinson/dev=
/hadoop-common/src/native=0A>>> -I/usr/lib/jvm/java-6-openjdk/include=0A>>>=
 -I/usr/lib/jvm/java-6-openjdk/include/linux=0A>>> -I/home/trobinson/dev/ha=
doop-common/src/native/src=0A>>> -Isrc/org/apache/hadoop/io/compress/zlib=
=0A>>> -Isrc/org/apache/hadoop/security -Isrc/org/apache/hadoop/io/nativeio=
/=0A>>> -g -Wall -fPIC -O2 -m32 -g -O2 -MT ZlibCompressor.lo -MD -MP -MF=0A=
>>> .deps/ZlibCompressor.Tpo -c -o ZlibCompressor.lo `test -f=0A>>> 'src/or=
g/apache/hadoop/io/compress/zlib/ZlibCompressor.c' || echo=0A>>> '/home/tro=
binson/dev/hadoop-common/src/native/'`src/org/apache/hadoop/io/compress/zli=
b/ZlibCompressor.c=0A>>> =A0 =A0 =A0[exec] libtool: compile: =A0gcc -DHAVE_=
CONFIG_H -I.=0A>>> -I/home/trobinson/dev/hadoop-common/src/native=0A>>> -I/=
usr/lib/jvm/java-6-openjdk/include=0A>>> -I/usr/lib/jvm/java-6-openjdk/incl=
ude/linux=0A>>> -I/home/trobinson/dev/hadoop-common/src/native/src=0A>>> -I=
src/org/apache/hadoop/io/compress/zlib=0A>>> -Isrc/org/apache/hadoop/securi=
ty -Isrc/org/apache/hadoop/io/nativeio/=0A>>> -g -Wall -fPIC -O2 -m32 -g -O=
2 -MT ZlibCompressor.lo -MD -MP -MF=0A>>> .deps/ZlibCompressor.Tpo -c=0A>>>=
 /home/trobinson/dev/hadoop-common/src/native/src/org/apache/hadoop/io/comp=
ress/zlib/ZlibCompressor.c=0A>>> =A0-fPIC -DPIC -o .libs/ZlibCompressor.o=
=0A>>> =A0 =A0 =A0[exec] make[1]: Leaving directory=0A>>> `/home/trobinson/=
dev/hadoop-common/build/native/Linux-arm-32'=0A>>> =A0 =A0 =A0[exec] cc1: e=
rror: unrecognized command line option "-m32"=0A>>> =A0 =A0 =A0[exec] make[=
1]: *** [ZlibCompressor.lo] Error 1=0A>>> =A0 =A0 =A0[exec] make: *** [all]=
 Error 2=0A>>>=0A>>> This closest issue I can find is=0A>>> https://issues.=
apache.org/jira/browse/HADOOP-6258=A0(Native compilation=0A>>> assumes gcc)=
, as well as other issues regarding where and how to=0A>>> specify -m32/64.=
 However, there doesn't seem to be a specific issue=0A>>> covering build fa=
ilure on systems using gcc where the gcc target does=0A>>> not support -m32=
/64 (such as ARM).=0A>>>=0A>>> I've attached a patch that disables specifyi=
ng -m$(JVM_DATA_MODEL)=0A>>> when $host_cpu starts with "arm". (For instanc=
e, host_cpu =3D armv7l for=0A>>> my system.) To any maintainers on this lis=
t, please let me know if=0A>>> you'd like me to open a new issue and/or att=
ach this patch to an=0A>>> issue.=0A>>>=0A>>> Thanks,=0A>>> Trevor=0A>>>=0A=
>>=0A>
--0-177622805-1306165605=:34385--