hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Wilcox <Tom.Wil...@cse.org.uk>
Subject Using Java Remote Method Invocation to make a UI for Hadoop
Date Mon, 09 Jan 2012 13:29:45 GMT
Hi,

We are trying to make a UI for our HBase + Hadoop applications.

Basically, what we want is a web front end that can be hosted on a different server and present
data from HBase, as well as allow us to launch Map-Reduce jobs from the web browser.

Our current approach is as follows:

We have a 6-node development cluster with each node running Scientific Linux in a VM with
the following configuration:

Node - Hostname - Daemons
--------------------------------------------
Node 1 - "namenode" - namenode, secondarynamenode, regionserver, hbase-master, zookeeper
Node 2 - "jobtracker" - jobtracker, zookeeper
Node 3 - "slave0" - zookeeper, datanode, tasktracker
Node 4 - "slave1" - datanode, tasktracker
Node 5 - "slave2" - datanode, tasktracker
Node 6 - "slave3" - datanode, tasktracker

We have a 7th scientific linux VM running a Grails web application called "BillyWeb". 

We have created several Map-Reduce applications that run on the namenode to process HBase
data and populate a results table. Currently, these MR apps are run from the command line
on namenode.

We use the HBase REST interface to query data from the results table and present it in an
AJAX-enabled web page. That is the reading part of the problem solved :)

Now we need to have a HTML page with a button, that when clicked will execute an MR app on
the namenode. This forms the writing part of the problem.

Currently, we are attempting to do this using Java's Remote Method Invocation (RMI) feature.
I have successfully created a Java client-server application where the server program runs
on "namenode" and the client runs on "BillyWeb" successfully. The client program calls a remote
object method to trigger the MR job on namenode and gets the result.

We are now in the process of integrating it into the Grails webapp, by adding the source code
and calling it using groovy taglibs. The only remaining issue we are wrestling with is the
need to modify the security policy of the web app to allow access to the client portions of
the source code to the remote server.

Eventually we hope to have an internal web site that you can visit in any web browser to view
custom-made visualisations of the data and execute parameterised MR jobs to process data stored
in HBase (amongst other places).

We would like to know your thoughts on this approach. In particular, this feels like a potentially
convoluted approach to building a front-end that may feature several redundant steps (are
we are reinventing too many wheels?). 

Is there an alternative approach that you can think of that might be more sensible? 

Can you think of any problems with the approach we are currently taking?

Thanks,
Tom
Mime
View raw message