airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sanjaya Medonsa <sanjaya...@gmail.com>
Subject Re: Proposal To Use Apache OODT products as input to Apache Airavata Workflows and staging product files into node where execution happens
Date Wed, 20 Feb 2013 18:29:38 GMT
Thanks Chris! As you suggested previously, I am looking into CAS-PGE and
plan is to reuse the same code. I have looked at PGETaskInstance where most
of these pre/post task execution implementation resides. I believe
FileManageFileStager class should be able to reuse  easily. First I'll
focus on the input fileStaging and then I am planning to focus on ingesting
products and updating metadata as post execution task.

Best Regards,
Sanjaya

On Wed, Feb 20, 2013 at 9:00 PM, Mattmann, Chris A (388J) <
chris.a.mattmann@jpl.nasa.gov> wrote:

> Hi Sanjaya,
>
> I would seriously recommend looking at OODT CAS-PGE, which already does
> file staging, and connects to the file manager using queries. You could
> wrap or sub-class CAS-PGE in Airavata and I think avoid rewriting a lot of
> the existing FM to WM to RM and crawl infrastructure from OODT.
>
> Cheers,
> Chris
>
> On 2/19/13 10:52 AM, "Sanjaya Medonsa" <sanjayamrt@gmail.com> wrote:
>
> >Thanks Lahiru! I have gone through the test classes and the classes in
> >package org.apache.airavata.gfac. It was really helpful to understand the
> >new architecture. I have listed down my approach based on new architecture
> >to use Apache OODT products as an input to Airavata.
> >         1. Introduce new Data Type to represent Apache OODT Product as an
> >DataType. Basically new DataType is added into the GFacParameterTypes.xsd.
> >         2. With new Architecture In Handlers and Out Handlers replaces
> >the
> >Pre/Post execution chains in old architecture. For the moment I am
> >focusing
> >on using Apache OODT product ID or file path as an input and stage the
> >file
> >(product) into host where actual execution happens. File staging requires
> >to retrieve product from a File Manager server to the host where execution
> >occurs. File staging can be implemented as an* In Handler* and needs to be
> >configured as a new item in the list of configured In Handlers.
> >         3. Handler should first verify the input parameter types listed
> >in
> >Service Description of the Application context of the
> >*JobExecutionContext*.
> >If input type matches the new parameter type, in handler stage file into
> >host machine using Apache OODT file manager component. Corresponding input
> >value can be retrieved from *In MessageContex*t. If a parameter type in
> >MessageContext matches the new input type, then corresponding value is the
> >id or file path to product managed by Apache OODT File Manager server.
> >
> >Best Regards,
> >Sanjaya
> >
> >On Tue, Feb 19, 2013 at 1:29 AM, Lahiru Gunathilake
> ><glahiru@gmail.com>wrote:
> >
> >> Hi Sanjaya,
> >>
> >> If you want to understand the new architecture by looking in to the
> >>code,
> >> please just refer the package org.apache.airavata.gfac, do not refer
> >>any of
> >> the classes in org.apache.airavata.core.gfac.
> >>
> >> Best place to start with is referring is from test classes
> >> (LocalProviderTest,GramProviderTest), from tehre you can start looking
> >>in
> >> to GFacAPI class and see how to execution is flawing.
> >>
> >> if you have further questions please post on the list and more than
> >>happy
> >> to help. I will be doing some documentation about the architecture,
> >>once I
> >> am done, will post in to the list. And we will be having an architecture
> >> review this week, so please watch the mailing list, if possible please
> >>try
> >> to join us.
> >>
> >> Regards
> >> Lahiru
> >>
> >> On Mon, Feb 18, 2013 at 1:25 PM, Sanjaya Medonsa <sanjayamrt@gmail.com
> >> >wrote:
> >>
> >> > Thanks Suresh and Chris! It seems I am moving on the correct path. I
> >>have
> >> > followed the email thread on improved GFac architecture. Though I am
> >>not
> >> > entirely clear on the improved GFac architecture, proposed integration
> >> with
> >> > OODT is primarily based on the GFac extension, PreExecustionChain,
> >>which
> >> > has not been modified with the Architecture improvements (As per one
> >>of
> >> the
> >> > replies from Lahiru, output extension is supported with new
> >> Architecture. I
> >> > assume input extension is also supported).
> >> >
> >> > I have looked into provenance manager and related implementation.
> >>Still I
> >> > am unclear how Airavata support provenance aware work flow processing.
> >> >
> >> > Best Regards,
> >> > Sanjaya
> >> >
> >> > On Mon, Feb 18, 2013 at 6:35 AM, Suresh Marru <smarru@apache.org>
> >>wrote:
> >> >
> >> > > Hi Sanjaya,
> >> > >
> >> > > This sounds very exciting. Both Airavata and OODT projects have good
> >> > > synergies and have been long looking for volunteers who can bridge
> >>them
> >> > > both. Please do not hesitate to ask any questions to either or both
> >>the
> >> > dev
> >> > > lists. The more engaged you are, you will find use cases and
> >>feedback
> >> > which
> >> > > should help your MSc project.
> >> > >
> >> > > Your plan sounds good. If you are following dev list, you may have
> >> > > noticed, the GFac architecture has been improved to properly support
> >> this
> >> > > kind of handler architecture.
> >> > >
> >> > > You may also want to look at Airavata Registry API which has
> >> organically
> >> > > emerged with the need, but can greatly benefit for OODT's provenance
> >> > > capabilities.
> >> > >
> >> > > Suresh
> >> > >
> >> > > On Feb 17, 2013, at 1:25 PM, "Mattmann, Chris A (388J)" <
> >> > > chris.a.mattmann@jpl.nasa.gov> wrote:
> >> > >
> >> > > > +1, sounds like a great idea Sanjaya!
> >> > > >
> >> > > > I'm copying dev@oodt so they can be in the conversation too.
> >> > > >
> >> > > > Cheers,
> >> > > > Chris
> >> > > >
> >> > > > On 2/17/13 10:22 AM, "Sanjaya Medonsa" <sanjayamrt@gmail.com>
> >>wrote:
> >> > > >
> >> > > >> Hi Dev Team,
> >> > > >> As I have posted previously, I am working on a Apache Airavata
+
> >> > Apache
> >> > > >> OODT integration as my MSc project. Following is one of the
> >>possible
> >> > > >> integration to leverage Apache OODT file management capability
> >>into
> >> > > Apache
> >> > > >> Airavata. Please review the proposal and let me know your
> >>feedback.
> >> > > >>
> >> > > >> Proposal To Use Apache OODT products as input to Apache Airavata
> >> > > Workflows
> >> > > >> and staging product files into node where execution happens
> >> > > >>
> >> > >
> >> >
> >>
> >>=========================================================================
> >>=
> >> > > >> ==============================
> >> > > >> 1. Introduce "Apache OODT Product" as a new GFacParameterType.
> >>New
> >> > > "Apache
> >> > > >> OODT Product" input type can sepcify "Product ID" or "File
Path
> >>to
> >> > > >> Product"
> >> > > >> as an input to Apache Airavata workflows.
> >> > > >> 2. Introduce new PreExecuteChain to retrieve Apache OODT
Products
> >> from
> >> > > >> File
> >> > > >> Manager Server managed using Apache OODT.
> >> > > >> 1. Using Apache OODT File Manager componenet transfer Product
> >>from
> >> > > Server
> >> > > >> to input directory of the application as configured using
> >>XBaya-GUI
> >> > > under
> >> > > >> advanced configuration. (Here the assumption is that Products
are
> >> > > >> accesible
> >> > > >> through Apache OODT File Manager server)
> >> > > >> 2. Finally reset the input value to local file path. I think
we
> >>can
> >> > > remove
> >> > > >> the OODT Product parameter from invocation context and add
new
> >>file
> >> > > >> parameter with value set to 'local path of the transferred
> >> product'. I
> >> > > am
> >> > > >> not quite sure what are the implications of changing input
> >>parameter
> >> > > type
> >> > > >> during the execution.
> >> > > >>
> >> > > >> Similar approach has been implemented for GridFTP and HTTP.
> >> > > >>
> >> > > >> Best Regards,
> >> > > >> Sanjaya
> >> > > >
> >> > >
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> System Analyst Programmer
> >> PTI Lab
> >> Indiana University
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message