hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Bockelman <bbock...@cse.unl.edu>
Subject Re: Adding a new platform support to Hadoop
Date Thu, 17 Nov 2011 15:26:22 GMT
Hi Amir,

The best person to engage is Allen Wittenauer, who has "fought the good fight" on this previously.
 It's a bit of an uphill battle, as the primary development platform is definitely Linux /
x86-64.  I saw him close a bunch of portability-related tickets a few weeks ago, which made
me a little sad.

You didn't say whether you want to only switch the processor, or the processor and the platform
(Linux).  I'll talk like you want to do both.

There's three layers of support to think of:
1) All the Java code and surrounding wrappers.  Likely the Java is very portable; the surrounding
wrappers have an occasional Linux-ism.  Easy-to-Medium difficulty - more about finding all
the little quirks and smoothing them.
2) The native library.  These are very skewed toward linux-specific optimizations, but I think
are really required to get working to get Really Great Performance.  Medium-to-hard difficulty
- you need a broad POSIX and Linux understanding to convert things.
3) The optimizations in the Java code.  Some performance tunings are made based on Oracle
Java on Linux on x86-64.  You need a deep expert in Hadoop, system architectures, and Java
to understand what was changed, why it worked better on the target platform, and how the optimization
will affect your target platform.  Extremely hard difficulty.

Brian

On Nov 17, 2011, at 9:02 AM, Amir Sanjar wrote:

> Is there any specific development, build, and packaging guidelines to add
> support for a new hardware platform, in this case PPC64, to hadoop?
> 
> Best Regards
> Amir Sanjar
> 
> Linux System Management Architect and Lead
> IBM Senior Software Engineer
> Phone# 512-286-8393
> Fax#      512-838-8858


Mime
View raw message