hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jon Brisbin <jon.bris...@npcinternational.com>
Subject Hadoop on Solaris 10 and in OSGi bundles
Date Thu, 06 Nov 2008 14:13:14 GMT
Has anyone got Hadoop and/or HBase to OSGify? I Googled for a half- 
hour on this topic and there's almost no mention of OSGi and Hadoop  
(one JIRA ticket that's more a tutorial on writing Hadoop patches than  
information about the problem). There are plenty of issues with  
classloading, I'm sure, but at the moment, I'm not running remote  
cluster tasks inside OSGi, just trying to write files into HDFS and  
HBase and submit jobs. I can't get past this first problem, though, so  
I haven't been able to make much progress.

I'm trying to evaluate whether SpringSource dm Server will work for us  
to replace JBoss. I was planning on implementing a Hadoop cluster for  
back-end services rather than trying to get more servers to do load- 
balancing (the work for various pages is getting bigger and bigger all  
the time). I deploy in Java 6 on Solaris 10 x86.

I was able to get plain-jane Hadoop 0.18.1 running on Solaris 10 after  
overriding all the 0.0.0.0 IP addresses in the default file with $ 
{hadoop.bind.address} and specifying -Dhadoop.bind.address=1.2.3.4 in  
hadoop-env.sh (it was throwing NPEs before I did that). On Solaris 10,  
whoami is in /usr/ucb, which isn't in the PATH by default, so the docs  
should say you need to add /usr/ucb to the the path if you're using  
Solaris (which I'm assuming not many people are doing since there's  
ZERO documentation for the platform).

I couldn't get the native libraries to compile at all (make complains  
about macros because the Solaris gmake is just a shill for BSD make  
in /usr/ccs/bin).

+-( /opt/hadoop-0.18.1/src/native ):# gmake
make  all-recursive
make: Fatal error in reader: Makefile, line 647: Badly formed macro  
assignment
Current working directory /opt/hadoop-0.18.1/src/native
gmake: *** [all] Error 1

Forget about using the Sun Studio compiler.

But the biggest bummer was trying to get bnd ( http://www.aqute.biz/Code/Bnd 
  ) to create an OSGi bundle from Hadoop (which I was able to do with  
ZooKeeper...clients are able to connect to ZooKeeper server fine in a  
Spring app on SpringSource dm/Equinox). I was just wanting to use the  
client functionality from the OSGi container, not embed a server. bnd  
gets confused with the "webapps" folder (among other things). I got it  
to ignore that for Import-Package and Export-Package headers, but  
Equinox is complaining that the manifest is screwed up. Bottom line  
seems to be that I can't use bnd to create an OSGi bundle for Hadoop  
0.18.1 like I did with ZooKeeper. Maybe I can create new bundles using  
eclilpse PDE, then import the source code, let eclipse compile  
everything (cross fingers that most dependencies are either already  
OSGified or can be OSGified with bnd), then export an OSGi bundle for  
the various pieces. Not sure if anyone's tried this yet.

A SpringSource-compatible bundle would put webapps in a bundle with a  
"web" personality and ideally there'd be a segregation of packages  
into client and server bundles (there looks to be other packages in  
there which would probably have to go into their own bundle, as well).  
In fact, it would be awesome if the whole thing could be dropped into  
Equinox and embedded into a Spring application. At the very least I'd  
like to be able to connect to HDFS without resorting to stuffing the  
Hadoop jar into my bundle. Extra-special functionality would include  
submitting Hadoop jobs and reading HBase tables and the like. Extra- 
extra special functionality would be deploying jobs and tasks inside  
Equinox on the cluster.

Thanks!

Jon Brisbin
Portal Webmaster
NPC International, Inc.


Mime
View raw message