oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lahiru Gunathilake <glah...@gmail.com>
Subject Re: Proposal To Use Apache OODT products as input to Apache Airavata Workflows and staging product files into node where execution happens
Date Tue, 19 Feb 2013 19:58:49 GMT
Hi Sanjaya,

This looks like a clean approach for me.

Regards
Lahiru

On Tue, Feb 19, 2013 at 1:52 PM, Sanjaya Medonsa <sanjayamrt@gmail.com>wrote:

> Thanks Lahiru! I have gone through the test classes and the classes in
> package org.apache.airavata.gfac. It was really helpful to understand the
> new architecture. I have listed down my approach based on new architecture
> to use Apache OODT products as an input to Airavata.
>          1. Introduce new Data Type to represent Apache OODT Product as an
> DataType. Basically new DataType is added into the GFacParameterTypes.xsd.
>          2. With new Architecture In Handlers and Out Handlers replaces the
> Pre/Post execution chains in old architecture. For the moment I am focusing
> on using Apache OODT product ID or file path as an input and stage the file
> (product) into host where actual execution happens. File staging requires
> to retrieve product from a File Manager server to the host where execution
> occurs. File staging can be implemented as an* In Handler* and needs to be
> configured as a new item in the list of configured In Handlers.
>          3. Handler should first verify the input parameter types listed in
> Service Description of the Application context of the
> *JobExecutionContext*.
> If input type matches the new parameter type, in handler stage file into
> host machine using Apache OODT file manager component. Corresponding input
> value can be retrieved from *In MessageContex*t. If a parameter type in
> MessageContext matches the new input type, then corresponding value is the
> id or file path to product managed by Apache OODT File Manager server.
>
> Best Regards,
> Sanjaya
>
> On Tue, Feb 19, 2013 at 1:29 AM, Lahiru Gunathilake <glahiru@gmail.com
> >wrote:
>
> > Hi Sanjaya,
> >
> > If you want to understand the new architecture by looking in to the code,
> > please just refer the package org.apache.airavata.gfac, do not refer any
> of
> > the classes in org.apache.airavata.core.gfac.
> >
> > Best place to start with is referring is from test classes
> > (LocalProviderTest,GramProviderTest), from tehre you can start looking in
> > to GFacAPI class and see how to execution is flawing.
> >
> > if you have further questions please post on the list and more than happy
> > to help. I will be doing some documentation about the architecture, once
> I
> > am done, will post in to the list. And we will be having an architecture
> > review this week, so please watch the mailing list, if possible please
> try
> > to join us.
> >
> > Regards
> > Lahiru
> >
> > On Mon, Feb 18, 2013 at 1:25 PM, Sanjaya Medonsa <sanjayamrt@gmail.com
> > >wrote:
> >
> > > Thanks Suresh and Chris! It seems I am moving on the correct path. I
> have
> > > followed the email thread on improved GFac architecture. Though I am
> not
> > > entirely clear on the improved GFac architecture, proposed integration
> > with
> > > OODT is primarily based on the GFac extension, PreExecustionChain,
> which
> > > has not been modified with the Architecture improvements (As per one of
> > the
> > > replies from Lahiru, output extension is supported with new
> > Architecture. I
> > > assume input extension is also supported).
> > >
> > > I have looked into provenance manager and related implementation.
> Still I
> > > am unclear how Airavata support provenance aware work flow processing.
> > >
> > > Best Regards,
> > > Sanjaya
> > >
> > > On Mon, Feb 18, 2013 at 6:35 AM, Suresh Marru <smarru@apache.org>
> wrote:
> > >
> > > > Hi Sanjaya,
> > > >
> > > > This sounds very exciting. Both Airavata and OODT projects have good
> > > > synergies and have been long looking for volunteers who can bridge
> them
> > > > both. Please do not hesitate to ask any questions to either or both
> the
> > > dev
> > > > lists. The more engaged you are, you will find use cases and feedback
> > > which
> > > > should help your MSc project.
> > > >
> > > > Your plan sounds good. If you are following dev list, you may have
> > > > noticed, the GFac architecture has been improved to properly support
> > this
> > > > kind of handler architecture.
> > > >
> > > > You may also want to look at Airavata Registry API which has
> > organically
> > > > emerged with the need, but can greatly benefit for OODT's provenance
> > > > capabilities.
> > > >
> > > > Suresh
> > > >
> > > > On Feb 17, 2013, at 1:25 PM, "Mattmann, Chris A (388J)" <
> > > > chris.a.mattmann@jpl.nasa.gov> wrote:
> > > >
> > > > > +1, sounds like a great idea Sanjaya!
> > > > >
> > > > > I'm copying dev@oodt so they can be in the conversation too.
> > > > >
> > > > > Cheers,
> > > > > Chris
> > > > >
> > > > > On 2/17/13 10:22 AM, "Sanjaya Medonsa" <sanjayamrt@gmail.com>
> wrote:
> > > > >
> > > > >> Hi Dev Team,
> > > > >> As I have posted previously, I am working on a Apache Airavata
+
> > > Apache
> > > > >> OODT integration as my MSc project. Following is one of the
> possible
> > > > >> integration to leverage Apache OODT file management capability
> into
> > > > Apache
> > > > >> Airavata. Please review the proposal and let me know your
> feedback.
> > > > >>
> > > > >> Proposal To Use Apache OODT products as input to Apache Airavata
> > > > Workflows
> > > > >> and staging product files into node where execution happens
> > > > >>
> > > >
> > >
> >
> ==========================================================================
> > > > >> ==============================
> > > > >> 1. Introduce "Apache OODT Product" as a new GFacParameterType.
New
> > > > "Apache
> > > > >> OODT Product" input type can sepcify "Product ID" or "File Path
to
> > > > >> Product"
> > > > >> as an input to Apache Airavata workflows.
> > > > >> 2. Introduce new PreExecuteChain to retrieve Apache OODT Products
> > from
> > > > >> File
> > > > >> Manager Server managed using Apache OODT.
> > > > >> 1. Using Apache OODT File Manager componenet transfer Product
from
> > > > Server
> > > > >> to input directory of the application as configured using
> XBaya-GUI
> > > > under
> > > > >> advanced configuration. (Here the assumption is that Products
are
> > > > >> accesible
> > > > >> through Apache OODT File Manager server)
> > > > >> 2. Finally reset the input value to local file path. I think
we
> can
> > > > remove
> > > > >> the OODT Product parameter from invocation context and add new
> file
> > > > >> parameter with value set to 'local path of the transferred
> > product'. I
> > > > am
> > > > >> not quite sure what are the implications of changing input
> parameter
> > > > type
> > > > >> during the execution.
> > > > >>
> > > > >> Similar approach has been implemented for GridFTP and HTTP.
> > > > >>
> > > > >> Best Regards,
> > > > >> Sanjaya
> > > > >
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > System Analyst Programmer
> > PTI Lab
> > Indiana University
> >
>



-- 
System Analyst Programmer
PTI Lab
Indiana University

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message