oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sanjaya Medonsa <sanjaya...@gmail.com>
Subject Re: Proposal To Use Apache OODT products as input to Apache Airavata Workflows and staging product files into node where execution happens
Date Fri, 22 Feb 2013 09:54:50 GMT
Thanks Chris! As you suggested best way to reuse the PgeTaskInstance is to
sub-class, as it contains several protected methods which are very useful
for integrating Apache OODT with workflow execution.

Best Regards,
Sanjaya

On Fri, Feb 22, 2013 at 9:22 AM, Mattmann, Chris A (388J) <
chris.a.mattmann@jpl.nasa.gov> wrote:

> Hi Sanjaya,
>
> Great. You may simply want to extend (sub-class) PgeTaskInstance, and
> create one for Airavata?
>
> Cheers,
> Chris
>
> On 2/20/13 10:29 AM, "Sanjaya Medonsa" <sanjayamrt@gmail.com> wrote:
>
> >Thanks Chris! As you suggested previously, I am looking into CAS-PGE and
> >plan is to reuse the same code. I have looked at PGETaskInstance where
> >most
> >of these pre/post task execution implementation resides. I believe
> >FileManageFileStager class should be able to reuse  easily. First I'll
> >focus on the input fileStaging and then I am planning to focus on
> >ingesting
> >products and updating metadata as post execution task.
> >
> >Best Regards,
> >Sanjaya
> >
> >On Wed, Feb 20, 2013 at 9:00 PM, Mattmann, Chris A (388J) <
> >chris.a.mattmann@jpl.nasa.gov> wrote:
> >
> >> Hi Sanjaya,
> >>
> >> I would seriously recommend looking at OODT CAS-PGE, which already does
> >> file staging, and connects to the file manager using queries. You could
> >> wrap or sub-class CAS-PGE in Airavata and I think avoid rewriting a lot
> >>of
> >> the existing FM to WM to RM and crawl infrastructure from OODT.
> >>
> >> Cheers,
> >> Chris
> >>
> >> On 2/19/13 10:52 AM, "Sanjaya Medonsa" <sanjayamrt@gmail.com> wrote:
> >>
> >> >Thanks Lahiru! I have gone through the test classes and the classes in
> >> >package org.apache.airavata.gfac. It was really helpful to understand
> >>the
> >> >new architecture. I have listed down my approach based on new
> >>architecture
> >> >to use Apache OODT products as an input to Airavata.
> >> >         1. Introduce new Data Type to represent Apache OODT Product
> >>as an
> >> >DataType. Basically new DataType is added into the
> >>GFacParameterTypes.xsd.
> >> >         2. With new Architecture In Handlers and Out Handlers replaces
> >> >the
> >> >Pre/Post execution chains in old architecture. For the moment I am
> >> >focusing
> >> >on using Apache OODT product ID or file path as an input and stage the
> >> >file
> >> >(product) into host where actual execution happens. File staging
> >>requires
> >> >to retrieve product from a File Manager server to the host where
> >>execution
> >> >occurs. File staging can be implemented as an* In Handler* and needs
> >>to be
> >> >configured as a new item in the list of configured In Handlers.
> >> >         3. Handler should first verify the input parameter types
> >>listed
> >> >in
> >> >Service Description of the Application context of the
> >> >*JobExecutionContext*.
> >> >If input type matches the new parameter type, in handler stage file
> >>into
> >> >host machine using Apache OODT file manager component. Corresponding
> >>input
> >> >value can be retrieved from *In MessageContex*t. If a parameter type in
> >> >MessageContext matches the new input type, then corresponding value is
> >>the
> >> >id or file path to product managed by Apache OODT File Manager server.
> >> >
> >> >Best Regards,
> >> >Sanjaya
> >> >
> >> >On Tue, Feb 19, 2013 at 1:29 AM, Lahiru Gunathilake
> >> ><glahiru@gmail.com>wrote:
> >> >
> >> >> Hi Sanjaya,
> >> >>
> >> >> If you want to understand the new architecture by looking in to the
> >> >>code,
> >> >> please just refer the package org.apache.airavata.gfac, do not refer
> >> >>any of
> >> >> the classes in org.apache.airavata.core.gfac.
> >> >>
> >> >> Best place to start with is referring is from test classes
> >> >> (LocalProviderTest,GramProviderTest), from tehre you can start
> >>looking
> >> >>in
> >> >> to GFacAPI class and see how to execution is flawing.
> >> >>
> >> >> if you have further questions please post on the list and more than
> >> >>happy
> >> >> to help. I will be doing some documentation about the architecture,
> >> >>once I
> >> >> am done, will post in to the list. And we will be having an
> >>architecture
> >> >> review this week, so please watch the mailing list, if possible
> >>please
> >> >>try
> >> >> to join us.
> >> >>
> >> >> Regards
> >> >> Lahiru
> >> >>
> >> >> On Mon, Feb 18, 2013 at 1:25 PM, Sanjaya Medonsa
> >><sanjayamrt@gmail.com
> >> >> >wrote:
> >> >>
> >> >> > Thanks Suresh and Chris! It seems I am moving on the correct path.
> >>I
> >> >>have
> >> >> > followed the email thread on improved GFac architecture. Though
I
> >>am
> >> >>not
> >> >> > entirely clear on the improved GFac architecture, proposed
> >>integration
> >> >> with
> >> >> > OODT is primarily based on the GFac extension, PreExecustionChain,
> >> >>which
> >> >> > has not been modified with the Architecture improvements (As per
> >>one
> >> >>of
> >> >> the
> >> >> > replies from Lahiru, output extension is supported with new
> >> >> Architecture. I
> >> >> > assume input extension is also supported).
> >> >> >
> >> >> > I have looked into provenance manager and related implementation.
> >> >>Still I
> >> >> > am unclear how Airavata support provenance aware work flow
> >>processing.
> >> >> >
> >> >> > Best Regards,
> >> >> > Sanjaya
> >> >> >
> >> >> > On Mon, Feb 18, 2013 at 6:35 AM, Suresh Marru <smarru@apache.org>
> >> >>wrote:
> >> >> >
> >> >> > > Hi Sanjaya,
> >> >> > >
> >> >> > > This sounds very exciting. Both Airavata and OODT projects
have
> >>good
> >> >> > > synergies and have been long looking for volunteers who can
> >>bridge
> >> >>them
> >> >> > > both. Please do not hesitate to ask any questions to either
or
> >>both
> >> >>the
> >> >> > dev
> >> >> > > lists. The more engaged you are, you will find use cases
and
> >> >>feedback
> >> >> > which
> >> >> > > should help your MSc project.
> >> >> > >
> >> >> > > Your plan sounds good. If you are following dev list, you
may
> >>have
> >> >> > > noticed, the GFac architecture has been improved to properly
> >>support
> >> >> this
> >> >> > > kind of handler architecture.
> >> >> > >
> >> >> > > You may also want to look at Airavata Registry API which
has
> >> >> organically
> >> >> > > emerged with the need, but can greatly benefit for OODT's
> >>provenance
> >> >> > > capabilities.
> >> >> > >
> >> >> > > Suresh
> >> >> > >
> >> >> > > On Feb 17, 2013, at 1:25 PM, "Mattmann, Chris A (388J)" <
> >> >> > > chris.a.mattmann@jpl.nasa.gov> wrote:
> >> >> > >
> >> >> > > > +1, sounds like a great idea Sanjaya!
> >> >> > > >
> >> >> > > > I'm copying dev@oodt so they can be in the conversation
too.
> >> >> > > >
> >> >> > > > Cheers,
> >> >> > > > Chris
> >> >> > > >
> >> >> > > > On 2/17/13 10:22 AM, "Sanjaya Medonsa" <sanjayamrt@gmail.com>
> >> >>wrote:
> >> >> > > >
> >> >> > > >> Hi Dev Team,
> >> >> > > >> As I have posted previously, I am working on a Apache
> >>Airavata +
> >> >> > Apache
> >> >> > > >> OODT integration as my MSc project. Following is
one of the
> >> >>possible
> >> >> > > >> integration to leverage Apache OODT file management
capability
> >> >>into
> >> >> > > Apache
> >> >> > > >> Airavata. Please review the proposal and let me
know your
> >> >>feedback.
> >> >> > > >>
> >> >> > > >> Proposal To Use Apache OODT products as input to
Apache
> >>Airavata
> >> >> > > Workflows
> >> >> > > >> and staging product files into node where execution
happens
> >> >> > > >>
> >> >> > >
> >> >> >
> >> >>
> >>
> >>>>=======================================================================
> >>>>==
> >> >>=
> >> >> > > >> ==============================
> >> >> > > >> 1. Introduce "Apache OODT Product" as a new GFacParameterType.
> >> >>New
> >> >> > > "Apache
> >> >> > > >> OODT Product" input type can sepcify "Product ID"
or "File
> >>Path
> >> >>to
> >> >> > > >> Product"
> >> >> > > >> as an input to Apache Airavata workflows.
> >> >> > > >> 2. Introduce new PreExecuteChain to retrieve Apache
OODT
> >>Products
> >> >> from
> >> >> > > >> File
> >> >> > > >> Manager Server managed using Apache OODT.
> >> >> > > >> 1. Using Apache OODT File Manager componenet transfer
Product
> >> >>from
> >> >> > > Server
> >> >> > > >> to input directory of the application as configured
using
> >> >>XBaya-GUI
> >> >> > > under
> >> >> > > >> advanced configuration. (Here the assumption is
that Products
> >>are
> >> >> > > >> accesible
> >> >> > > >> through Apache OODT File Manager server)
> >> >> > > >> 2. Finally reset the input value to local file path.
I think
> >>we
> >> >>can
> >> >> > > remove
> >> >> > > >> the OODT Product parameter from invocation context
and add new
> >> >>file
> >> >> > > >> parameter with value set to 'local path of the transferred
> >> >> product'. I
> >> >> > > am
> >> >> > > >> not quite sure what are the implications of changing
input
> >> >>parameter
> >> >> > > type
> >> >> > > >> during the execution.
> >> >> > > >>
> >> >> > > >> Similar approach has been implemented for GridFTP
and HTTP.
> >> >> > > >>
> >> >> > > >> Best Regards,
> >> >> > > >> Sanjaya
> >> >> > > >
> >> >> > >
> >> >> > >
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> System Analyst Programmer
> >> >> PTI Lab
> >> >> Indiana University
> >> >>
> >>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message