hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Wilcox <hungry...@gmail.com>
Subject Using Java Remote Method Invocation to make a UI for Hadoop
Date Mon, 09 Jan 2012 12:03:10 GMT
Hi,

We are trying to make a UI for our HBase + Hadoop applications.

Basically, what we want is a web front end that can be hosted on a
different server and present data from HBase, as well as allow us to launch
Map-Reduce jobs from the web browser.

Our current approach is as follows:

We have a 6-node development cluster with each node running Scientific
Linux in a VM with the following configuration:

Node - Hostname - Daemons
--------------------------------------------
Node 1 - "namenode" - namenode, secondarynamenode, regionserver,
hbase-master, zookeeper
Node 2 - "jobtracker" - jobtracker, zookeeper
Node 3 - "slave0" - zookeeper, datanode, tasktracker
Node 4 - "slave1" - datanode, tasktracker
Node 5 - "slave2" - datanode, tasktracker
Node 6 - "slave3" - datanode, tasktracker

We have a 7th scientific linux VM running a Grails web application called
"BillyWeb".

We have created several Map-Reduce applications that run on the namenode to
process HBase data and populate a results table. Currently, these MR apps
are run from the command line on namenode.

We use the HBase REST interface to query data from the results table and
present it in an AJAX-enabled web page. That is the reading part of the
problem solved :)

Now we need to have a HTML page with a button, that when clicked will
execute an MR app on the namenode. This forms the writing part of the
problem.

Currently, we are attempting to do this using Java's Remote Method
Invocation (RMI) feature. I have successfully created a Java client-server
application where the server program runs on "namenode" and the client runs
on "BillyWeb" successfully. The client program calls a remote object method
to trigger the MR job on namenode and gets the result.

We are now in the process of integrating it into the Grails webapp, by
adding the source code and calling it using groovy taglibs. The only
remaining issue we are wrestling with is the need to modify the security
policy of the web app to allow access to the client portions of the source
code to the remote server.

Eventually we hope to have an internal web site that you can visit in any
web browser to view custom-made visualisations of the data and execute
parameterised MR jobs to process data stored in HBase (amongst other
places).

We would like to know your thoughts on this approach. In particular, this
feels like a potentially convoluted approach to building a front-end that
may feature several redundant steps (are we are reinventing too many
wheels?).

Is there an alternative approach that you can think of that might be more
sensible?

Can you think of any problems with the approach we are currently taking?

Thanks,
Tom

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message