oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suresh Marru <sma...@apache.org>
Subject Re: Proposal To Use Apache OODT products as input to Apache Airavata Workflows and staging product files into node where execution happens
Date Wed, 20 Feb 2013 02:33:09 GMT
On Feb 19, 2013, at 1:52 PM, Sanjaya Medonsa <sanjayamrt@gmail.com> wrote:

> Thanks Lahiru! I have gone through the test classes and the classes in
> package org.apache.airavata.gfac. It was really helpful to understand the
> new architecture. I have listed down my approach based on new architecture
> to use Apache OODT products as an input to Airavata.
>         1. Introduce new Data Type to represent Apache OODT Product as an
> DataType. Basically new DataType is added into the GFacParameterTypes.xsd.
>         2. With new Architecture In Handlers and Out Handlers replaces the
> Pre/Post execution chains in old architecture. For the moment I am focusing
> on using Apache OODT product ID or file path as an input and stage the file
> (product) into host where actual execution happens. File staging requires
> to retrieve product from a File Manager server to the host where execution
> occurs. File staging can be implemented as an* In Handler* and needs to be
> configured as a new item in the list of configured In Handlers.
>         3. Handler should first verify the input parameter types listed in
> Service Description of the Application context of the *JobExecutionContext*.
> If input type matches the new parameter type, in handler stage file into
> host machine using Apache OODT file manager component. Corresponding input
> value can be retrieved from *In MessageContex*t. If a parameter type in
> MessageContext matches the new input type, then corresponding value is the
> id or file path to product managed by Apache OODT File Manager server.

Hi Sanjaya,

+ 1 to this plan. You are right on!! great job in dissecting the code and identifying where
you need to plug in. Since you are bridging both the projects, these kind of details will
greatly help every one understand what you are doing and offer early help. 


> Best Regards,
> Sanjaya
> On Tue, Feb 19, 2013 at 1:29 AM, Lahiru Gunathilake <glahiru@gmail.com>wrote:
>> Hi Sanjaya,
>> If you want to understand the new architecture by looking in to the code,
>> please just refer the package org.apache.airavata.gfac, do not refer any of
>> the classes in org.apache.airavata.core.gfac.
>> Best place to start with is referring is from test classes
>> (LocalProviderTest,GramProviderTest), from tehre you can start looking in
>> to GFacAPI class and see how to execution is flawing.
>> if you have further questions please post on the list and more than happy
>> to help. I will be doing some documentation about the architecture, once I
>> am done, will post in to the list. And we will be having an architecture
>> review this week, so please watch the mailing list, if possible please try
>> to join us.
>> Regards
>> Lahiru
>> On Mon, Feb 18, 2013 at 1:25 PM, Sanjaya Medonsa <sanjayamrt@gmail.com
>>> wrote:
>>> Thanks Suresh and Chris! It seems I am moving on the correct path. I have
>>> followed the email thread on improved GFac architecture. Though I am not
>>> entirely clear on the improved GFac architecture, proposed integration
>> with
>>> OODT is primarily based on the GFac extension, PreExecustionChain, which
>>> has not been modified with the Architecture improvements (As per one of
>> the
>>> replies from Lahiru, output extension is supported with new
>> Architecture. I
>>> assume input extension is also supported).
>>> I have looked into provenance manager and related implementation. Still I
>>> am unclear how Airavata support provenance aware work flow processing.
>>> Best Regards,
>>> Sanjaya
>>> On Mon, Feb 18, 2013 at 6:35 AM, Suresh Marru <smarru@apache.org> wrote:
>>>> Hi Sanjaya,
>>>> This sounds very exciting. Both Airavata and OODT projects have good
>>>> synergies and have been long looking for volunteers who can bridge them
>>>> both. Please do not hesitate to ask any questions to either or both the
>>> dev
>>>> lists. The more engaged you are, you will find use cases and feedback
>>> which
>>>> should help your MSc project.
>>>> Your plan sounds good. If you are following dev list, you may have
>>>> noticed, the GFac architecture has been improved to properly support
>> this
>>>> kind of handler architecture.
>>>> You may also want to look at Airavata Registry API which has
>> organically
>>>> emerged with the need, but can greatly benefit for OODT's provenance
>>>> capabilities.
>>>> Suresh
>>>> On Feb 17, 2013, at 1:25 PM, "Mattmann, Chris A (388J)" <
>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>> +1, sounds like a great idea Sanjaya!
>>>>> I'm copying dev@oodt so they can be in the conversation too.
>>>>> Cheers,
>>>>> Chris
>>>>> On 2/17/13 10:22 AM, "Sanjaya Medonsa" <sanjayamrt@gmail.com> wrote:
>>>>>> Hi Dev Team,
>>>>>> As I have posted previously, I am working on a Apache Airavata +
>>> Apache
>>>>>> OODT integration as my MSc project. Following is one of the possible
>>>>>> integration to leverage Apache OODT file management capability into
>>>> Apache
>>>>>> Airavata. Please review the proposal and let me know your feedback.
>>>>>> Proposal To Use Apache OODT products as input to Apache Airavata
>>>> Workflows
>>>>>> and staging product files into node where execution happens
>> ==========================================================================
>>>>>> ==============================
>>>>>> 1. Introduce "Apache OODT Product" as a new GFacParameterType. New
>>>> "Apache
>>>>>> OODT Product" input type can sepcify "Product ID" or "File Path to
>>>>>> Product"
>>>>>> as an input to Apache Airavata workflows.
>>>>>> 2. Introduce new PreExecuteChain to retrieve Apache OODT Products
>> from
>>>>>> File
>>>>>> Manager Server managed using Apache OODT.
>>>>>> 1. Using Apache OODT File Manager componenet transfer Product from
>>>> Server
>>>>>> to input directory of the application as configured using XBaya-GUI
>>>> under
>>>>>> advanced configuration. (Here the assumption is that Products are
>>>>>> accesible
>>>>>> through Apache OODT File Manager server)
>>>>>> 2. Finally reset the input value to local file path. I think we can
>>>> remove
>>>>>> the OODT Product parameter from invocation context and add new file
>>>>>> parameter with value set to 'local path of the transferred
>> product'. I
>>>> am
>>>>>> not quite sure what are the implications of changing input parameter
>>>> type
>>>>>> during the execution.
>>>>>> Similar approach has been implemented for GridFTP and HTTP.
>>>>>> Best Regards,
>>>>>> Sanjaya
>> --
>> System Analyst Programmer
>> PTI Lab
>> Indiana University

View raw message