airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mattmann, Chris A (388J)" <chris.a.mattm...@jpl.nasa.gov>
Subject Re: Proposal To Use Apache OODT products as input to Apache Airavata Workflows and staging product files into node where execution happens
Date Fri, 22 Feb 2013 03:52:43 GMT
Hi Sanjaya,

Great. You may simply want to extend (sub-class) PgeTaskInstance, and
create one for Airavata?

Cheers,
Chris

On 2/20/13 10:29 AM, "Sanjaya Medonsa" <sanjayamrt@gmail.com> wrote:

>Thanks Chris! As you suggested previously, I am looking into CAS-PGE and
>plan is to reuse the same code. I have looked at PGETaskInstance where
>most
>of these pre/post task execution implementation resides. I believe
>FileManageFileStager class should be able to reuse  easily. First I'll
>focus on the input fileStaging and then I am planning to focus on
>ingesting
>products and updating metadata as post execution task.
>
>Best Regards,
>Sanjaya
>
>On Wed, Feb 20, 2013 at 9:00 PM, Mattmann, Chris A (388J) <
>chris.a.mattmann@jpl.nasa.gov> wrote:
>
>> Hi Sanjaya,
>>
>> I would seriously recommend looking at OODT CAS-PGE, which already does
>> file staging, and connects to the file manager using queries. You could
>> wrap or sub-class CAS-PGE in Airavata and I think avoid rewriting a lot
>>of
>> the existing FM to WM to RM and crawl infrastructure from OODT.
>>
>> Cheers,
>> Chris
>>
>> On 2/19/13 10:52 AM, "Sanjaya Medonsa" <sanjayamrt@gmail.com> wrote:
>>
>> >Thanks Lahiru! I have gone through the test classes and the classes in
>> >package org.apache.airavata.gfac. It was really helpful to understand
>>the
>> >new architecture. I have listed down my approach based on new
>>architecture
>> >to use Apache OODT products as an input to Airavata.
>> >         1. Introduce new Data Type to represent Apache OODT Product
>>as an
>> >DataType. Basically new DataType is added into the
>>GFacParameterTypes.xsd.
>> >         2. With new Architecture In Handlers and Out Handlers replaces
>> >the
>> >Pre/Post execution chains in old architecture. For the moment I am
>> >focusing
>> >on using Apache OODT product ID or file path as an input and stage the
>> >file
>> >(product) into host where actual execution happens. File staging
>>requires
>> >to retrieve product from a File Manager server to the host where
>>execution
>> >occurs. File staging can be implemented as an* In Handler* and needs
>>to be
>> >configured as a new item in the list of configured In Handlers.
>> >         3. Handler should first verify the input parameter types
>>listed
>> >in
>> >Service Description of the Application context of the
>> >*JobExecutionContext*.
>> >If input type matches the new parameter type, in handler stage file
>>into
>> >host machine using Apache OODT file manager component. Corresponding
>>input
>> >value can be retrieved from *In MessageContex*t. If a parameter type in
>> >MessageContext matches the new input type, then corresponding value is
>>the
>> >id or file path to product managed by Apache OODT File Manager server.
>> >
>> >Best Regards,
>> >Sanjaya
>> >
>> >On Tue, Feb 19, 2013 at 1:29 AM, Lahiru Gunathilake
>> ><glahiru@gmail.com>wrote:
>> >
>> >> Hi Sanjaya,
>> >>
>> >> If you want to understand the new architecture by looking in to the
>> >>code,
>> >> please just refer the package org.apache.airavata.gfac, do not refer
>> >>any of
>> >> the classes in org.apache.airavata.core.gfac.
>> >>
>> >> Best place to start with is referring is from test classes
>> >> (LocalProviderTest,GramProviderTest), from tehre you can start
>>looking
>> >>in
>> >> to GFacAPI class and see how to execution is flawing.
>> >>
>> >> if you have further questions please post on the list and more than
>> >>happy
>> >> to help. I will be doing some documentation about the architecture,
>> >>once I
>> >> am done, will post in to the list. And we will be having an
>>architecture
>> >> review this week, so please watch the mailing list, if possible
>>please
>> >>try
>> >> to join us.
>> >>
>> >> Regards
>> >> Lahiru
>> >>
>> >> On Mon, Feb 18, 2013 at 1:25 PM, Sanjaya Medonsa
>><sanjayamrt@gmail.com
>> >> >wrote:
>> >>
>> >> > Thanks Suresh and Chris! It seems I am moving on the correct path.
>>I
>> >>have
>> >> > followed the email thread on improved GFac architecture. Though I
>>am
>> >>not
>> >> > entirely clear on the improved GFac architecture, proposed
>>integration
>> >> with
>> >> > OODT is primarily based on the GFac extension, PreExecustionChain,
>> >>which
>> >> > has not been modified with the Architecture improvements (As per
>>one
>> >>of
>> >> the
>> >> > replies from Lahiru, output extension is supported with new
>> >> Architecture. I
>> >> > assume input extension is also supported).
>> >> >
>> >> > I have looked into provenance manager and related implementation.
>> >>Still I
>> >> > am unclear how Airavata support provenance aware work flow
>>processing.
>> >> >
>> >> > Best Regards,
>> >> > Sanjaya
>> >> >
>> >> > On Mon, Feb 18, 2013 at 6:35 AM, Suresh Marru <smarru@apache.org>
>> >>wrote:
>> >> >
>> >> > > Hi Sanjaya,
>> >> > >
>> >> > > This sounds very exciting. Both Airavata and OODT projects have
>>good
>> >> > > synergies and have been long looking for volunteers who can
>>bridge
>> >>them
>> >> > > both. Please do not hesitate to ask any questions to either or
>>both
>> >>the
>> >> > dev
>> >> > > lists. The more engaged you are, you will find use cases and
>> >>feedback
>> >> > which
>> >> > > should help your MSc project.
>> >> > >
>> >> > > Your plan sounds good. If you are following dev list, you may
>>have
>> >> > > noticed, the GFac architecture has been improved to properly
>>support
>> >> this
>> >> > > kind of handler architecture.
>> >> > >
>> >> > > You may also want to look at Airavata Registry API which has
>> >> organically
>> >> > > emerged with the need, but can greatly benefit for OODT's
>>provenance
>> >> > > capabilities.
>> >> > >
>> >> > > Suresh
>> >> > >
>> >> > > On Feb 17, 2013, at 1:25 PM, "Mattmann, Chris A (388J)" <
>> >> > > chris.a.mattmann@jpl.nasa.gov> wrote:
>> >> > >
>> >> > > > +1, sounds like a great idea Sanjaya!
>> >> > > >
>> >> > > > I'm copying dev@oodt so they can be in the conversation too.
>> >> > > >
>> >> > > > Cheers,
>> >> > > > Chris
>> >> > > >
>> >> > > > On 2/17/13 10:22 AM, "Sanjaya Medonsa" <sanjayamrt@gmail.com>
>> >>wrote:
>> >> > > >
>> >> > > >> Hi Dev Team,
>> >> > > >> As I have posted previously, I am working on a Apache
>>Airavata +
>> >> > Apache
>> >> > > >> OODT integration as my MSc project. Following is one
of the
>> >>possible
>> >> > > >> integration to leverage Apache OODT file management capability
>> >>into
>> >> > > Apache
>> >> > > >> Airavata. Please review the proposal and let me know
your
>> >>feedback.
>> >> > > >>
>> >> > > >> Proposal To Use Apache OODT products as input to Apache
>>Airavata
>> >> > > Workflows
>> >> > > >> and staging product files into node where execution happens
>> >> > > >>
>> >> > >
>> >> >
>> >>
>> 
>>>>=======================================================================
>>>>==
>> >>=
>> >> > > >> ==============================
>> >> > > >> 1. Introduce "Apache OODT Product" as a new GFacParameterType.
>> >>New
>> >> > > "Apache
>> >> > > >> OODT Product" input type can sepcify "Product ID" or
"File
>>Path
>> >>to
>> >> > > >> Product"
>> >> > > >> as an input to Apache Airavata workflows.
>> >> > > >> 2. Introduce new PreExecuteChain to retrieve Apache OODT
>>Products
>> >> from
>> >> > > >> File
>> >> > > >> Manager Server managed using Apache OODT.
>> >> > > >> 1. Using Apache OODT File Manager componenet transfer
Product
>> >>from
>> >> > > Server
>> >> > > >> to input directory of the application as configured using
>> >>XBaya-GUI
>> >> > > under
>> >> > > >> advanced configuration. (Here the assumption is that
Products
>>are
>> >> > > >> accesible
>> >> > > >> through Apache OODT File Manager server)
>> >> > > >> 2. Finally reset the input value to local file path.
I think
>>we
>> >>can
>> >> > > remove
>> >> > > >> the OODT Product parameter from invocation context and
add new
>> >>file
>> >> > > >> parameter with value set to 'local path of the transferred
>> >> product'. I
>> >> > > am
>> >> > > >> not quite sure what are the implications of changing
input
>> >>parameter
>> >> > > type
>> >> > > >> during the execution.
>> >> > > >>
>> >> > > >> Similar approach has been implemented for GridFTP and
HTTP.
>> >> > > >>
>> >> > > >> Best Regards,
>> >> > > >> Sanjaya
>> >> > > >
>> >> > >
>> >> > >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> System Analyst Programmer
>> >> PTI Lab
>> >> Indiana University
>> >>
>>
>>


Mime
View raw message