hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@apache.org>
Subject Re: Start Hadoop env using JAVA or HADOOP APIs (InProcess)
Date Mon, 07 Dec 2009 13:30:31 GMT
samuellawrence wrote:
> Hai,
> I have to start the HADOOP environment using java code (inprocess). I would
> like to use the APIs to start it.  
> Could anyone please give me snippet or a link.

1. I've been starting/stopping Hadoop with SmartFrog, in JVM. Email me 
direct and I will point you at some of the code, though I have branched 
Hadoop (temporarily) to make it easier.

2. The MiniDFS and MiniMR clusters used in testing actually do this 

3. As a result of (1)  I know what the troublespots are if you are 
trying to run Hadoop in the VM of any other code
  * changes to the java security stuff in 0.21; incompatible with 
security managers
  * still a fair few singletons in the services
  * JSP under Jetty can be quirky and not always restart. This is not a 
hadoop-level bug, but something deep in Jasper, something probably 
related to JSP classloaders.

You can do this stuff in production, but you should consider having a 
separate VM for each service, and terminating the process when you are 
done with the specific node type. It's safer that way


View raw message