incubator-airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mattmann, Chris A (388J)" <chris.a.mattm...@jpl.nasa.gov>
Subject Re: Development Roadmap
Date Thu, 19 May 2011 05:00:22 GMT
Hi Suresh,

I've been meaning to respond to this, believe it or not, this is my first opportunity to!
:-)

Comments inline below:

> All of us clearly know what Airavata software is about in varying details,  but at the
same time I realize not every one of us on the list have a full understanding of the architecture
as a whole and sub-components. Along with inheriting the code donation, I suggest we focus
on bringing every one to speed by means of high level and low level architecture diagrams.
I will start a detailed email thread about this task. In short, currently the software assumes
understanding of e-Science in general and some details of Grid Computing. Our first focus
should be to bring the software to a level any java developer can understand and contribute.
Next the focus can be to make it easy for novice users.
> 
> I thought a good place to start might be to list out the high level goals and then focus
on the first goal with detailed JIRA tasks. I am assuming you will steer us with a orthogonal
roadmap to graduation. I hope I am not implying we need to meet the following goals to graduate,
because some of them are very open ended. Also, please note that Airavata may have some of
these features already, I am mainly categorizing so we will have a focused effort in testing,
re-writing or new implementations. 
> 
> Airavata high level feature list: 
> 
> Phase 1: Construct, Execute and monitor workflows from pre-deployed web services.

For workflow monitoring, check out the work we did in OODT in case you're interested. We recently
built some JAX-RS-based services for health monitoring and pedigree tracing: OODT-139 [1].


> The workflow enactment engine will be the inherent Airavata Workflow Interpreter. Register
command line applications as web services, construct and execute workflows with these application
services. The applications may run locally, on Grid enabled resources or by ssh'ing to a remote
resource. The client to test this phase workflows can be Airavata Workflow Client (XBaya)
running as a desktop application. 

Very cool! Is this code somewhere? I wonder if it would make sense to wrap the Airavata workflow
engine with the OODT workflow engine interface. Then OODT system users could benefit from
the monitoring and tool support apps and execute pre-built workflows on the Airavata engine.


I'd love to take a look at the Airvata WE, so a pointer would be great.

> 
> Phase 2: Execute all of phase 1 workflows on Apache ODE engine by generating and deploying
BPEL. Develop and deploy gadget interfaces to Apache Rave container to support application
registration, workflow submission and monitoring components. Support applications running
on virtual machine images to be deployed to Amazon EC2, EUCALYPTUS and similar infrastructure-as-a-service
cloud deployments. 

Very cool. Do you think it's worth engaging the folks in the Hadoop community? They've done
a done of work to execute tasks on EC2. Also it might be good to check out Apache Whirr [2],
which is trying to abstract job execution and resource management for different cloud stacks.

> 
> Phase 3:  Expand the compute resources to Elastic Map Reduce and Hadoop based executions.
Focus on the data and metadata catalog integration like Apache OODT. 

Sure. I'd just like to point out though that OODT isn't just data and metadata catalog integration.
In fact, it's got most of the workflow stuff (along with resource management, task wrapping
and execution and integration with Ganglia, and Torque/etc.) you've talked about above. Check
out these components:

http://oodt.apache.org/components/maven/workflow/
http://oodt.apache.org/components/maven/resource/

Also, this paper is worth checking out:

http://sunset.usc.edu/~mattmann/pubs/SMCIT09.pdf

> 
> I will stop here, to allow us to discuss the same. Once we narrow down on the high level
phase 1 goals, I will start a detailed discussion on where the code is now and the steps to
get to goal1.

For the phase 1 goal of monitoring, I can definitely help out with JAX-RS type services like
the ones from OODT-139 [1].

HTH,
Chris

[1] http://issues.apache.org/jira/browse/OODT-139
[2] http://incubator.apache.org/whirr/

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


Mime
View raw message