hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kovachev <dejan_kovac...@yahoo.com>
Subject Re: Hadoop on Solaris 10 and in OSGi bundles
Date Wed, 28 Apr 2010 16:30:50 GMT


we are trying to set up Hadoop to run on Solaris 10 within Containers.
However, we encounter many problems. 
Could you please write here down all the extra settings needed for running
Hadoop on Solaris?


Jon Brisbin-2 wrote:
> Has anyone got Hadoop and/or HBase to OSGify? I Googled for a half- 
> hour on this topic and there's almost no mention of OSGi and Hadoop  
> (one JIRA ticket that's more a tutorial on writing Hadoop patches than  
> information about the problem). There are plenty of issues with  
> classloading, I'm sure, but at the moment, I'm not running remote  
> cluster tasks inside OSGi, just trying to write files into HDFS and  
> HBase and submit jobs. I can't get past this first problem, though, so  
> I haven't been able to make much progress.
> I'm trying to evaluate whether SpringSource dm Server will work for us  
> to replace JBoss. I was planning on implementing a Hadoop cluster for  
> back-end services rather than trying to get more servers to do load- 
> balancing (the work for various pages is getting bigger and bigger all  
> the time). I deploy in Java 6 on Solaris 10 x86.
> I was able to get plain-jane Hadoop 0.18.1 running on Solaris 10 after  
> overriding all the IP addresses in the default file with $ 
> {hadoop.bind.address} and specifying -Dhadoop.bind.address= in  
> hadoop-env.sh (it was throwing NPEs before I did that). On Solaris 10,  
> whoami is in /usr/ucb, which isn't in the PATH by default, so the docs  
> should say you need to add /usr/ucb to the the path if you're using  
> Solaris (which I'm assuming not many people are doing since there's  
> ZERO documentation for the platform).
> I couldn't get the native libraries to compile at all (make complains  
> about macros because the Solaris gmake is just a shill for BSD make  
> in /usr/ccs/bin).
> +-( /opt/hadoop-0.18.1/src/native ):# gmake
> make  all-recursive
> make: Fatal error in reader: Makefile, line 647: Badly formed macro  
> assignment
> Current working directory /opt/hadoop-0.18.1/src/native
> gmake: *** [all] Error 1
> Forget about using the Sun Studio compiler.
> But the biggest bummer was trying to get bnd (
> http://www.aqute.biz/Code/Bnd 
>   ) to create an OSGi bundle from Hadoop (which I was able to do with  
> ZooKeeper...clients are able to connect to ZooKeeper server fine in a  
> Spring app on SpringSource dm/Equinox). I was just wanting to use the  
> client functionality from the OSGi container, not embed a server. bnd  
> gets confused with the "webapps" folder (among other things). I got it  
> to ignore that for Import-Package and Export-Package headers, but  
> Equinox is complaining that the manifest is screwed up. Bottom line  
> seems to be that I can't use bnd to create an OSGi bundle for Hadoop  
> 0.18.1 like I did with ZooKeeper. Maybe I can create new bundles using  
> eclilpse PDE, then import the source code, let eclipse compile  
> everything (cross fingers that most dependencies are either already  
> OSGified or can be OSGified with bnd), then export an OSGi bundle for  
> the various pieces. Not sure if anyone's tried this yet.
> A SpringSource-compatible bundle would put webapps in a bundle with a  
> "web" personality and ideally there'd be a segregation of packages  
> into client and server bundles (there looks to be other packages in  
> there which would probably have to go into their own bundle, as well).  
> In fact, it would be awesome if the whole thing could be dropped into  
> Equinox and embedded into a Spring application. At the very least I'd  
> like to be able to connect to HDFS without resorting to stuffing the  
> Hadoop jar into my bundle. Extra-special functionality would include  
> submitting Hadoop jobs and reading HBase tables and the like. Extra- 
> extra special functionality would be deploying jobs and tasks inside  
> Equinox on the cluster.
> Thanks!
> Jon Brisbin
> Portal Webmaster
> NPC International, Inc.

View this message in context: http://old.nabble.com/Hadoop-on-Solaris-10-and-in-OSGi-bundles-tp20361820p28391003.html
Sent from the Hadoop core-dev mailing list archive at Nabble.com.

View raw message