Mailing-List: contact dev-help@airavata.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airavata.apache.org
Received-SPF: pass (athena.apache.org: local policy)
From: "Mattmann, Chris A (388J)" <chris.a.mattmann@jpl.nasa.gov>
To: "dev@airavata.apache.org" <dev@airavata.apache.org>
CC: "dev@oodt.apache.org" <dev@oodt.apache.org>
Subject: Re: Proposal To Use Apache OODT products as input to Apache
 Airavata Workflows and staging product files into node where execution
 happens
Thread-Topic: Proposal To Use Apache OODT products as input to Apache
 Airavata Workflows and staging product files into node where execution
 happens
Thread-Index: 
 AQHODTvPNP0RpBHvB0SlY+IOOCKwhJh+XfkAgAD13wCAASJeAIAAGkyAgAF/mQCAANP5gA==
Date: Wed, 20 Feb 2013 15:30:55 +0000
Message-ID: <CD4A2C07.B7034%chris.a.mattmann@jpl.nasa.gov>
In-Reply-To: 
 <CAJEo2dFfOH6yGLZVUhmykdtVLt-dapdr3hDbtAOnH14FpECetQ@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
user-agent: Microsoft-MacOutlook/14.3.1.130117
Content-Type: text/plain; charset="us-ascii"
Content-ID: <A6989DE2C94FB242B2FED3B69EA343C2@ad.jpl>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hi Sanjaya,

I would seriously recommend looking at OODT CAS-PGE, which already does
file staging, and connects to the file manager using queries. You could
wrap or sub-class CAS-PGE in Airavata and I think avoid rewriting a lot of
the existing FM to WM to RM and crawl infrastructure from OODT.

Cheers,
Chris

On 2/19/13 10:52 AM, "Sanjaya Medonsa" <sanjayamrt@gmail.com> wrote:

>Thanks Lahiru! I have gone through the test classes and the classes in
>package org.apache.airavata.gfac. It was really helpful to understand the
>new architecture. I have listed down my approach based on new architecture
>to use Apache OODT products as an input to Airavata.
>         1. Introduce new Data Type to represent Apache OODT Product as an
>DataType. Basically new DataType is added into the GFacParameterTypes.xsd.
>         2. With new Architecture In Handlers and Out Handlers replaces
>the
>Pre/Post execution chains in old architecture. For the moment I am
>focusing
>on using Apache OODT product ID or file path as an input and stage the
>file
>(product) into host where actual execution happens. File staging requires
>to retrieve product from a File Manager server to the host where execution
>occurs. File staging can be implemented as an* In Handler* and needs to be
>configured as a new item in the list of configured In Handlers.
>         3. Handler should first verify the input parameter types listed
>in
>Service Description of the Application context of the
>*JobExecutionContext*.
>If input type matches the new parameter type, in handler stage file into
>host machine using Apache OODT file manager component. Corresponding input
>value can be retrieved from *In MessageContex*t. If a parameter type in
>MessageContext matches the new input type, then corresponding value is the
>id or file path to product managed by Apache OODT File Manager server.
>
>Best Regards,
>Sanjaya
>
>On Tue, Feb 19, 2013 at 1:29 AM, Lahiru Gunathilake
><glahiru@gmail.com>wrote:
>
>> Hi Sanjaya,
>>
>> If you want to understand the new architecture by looking in to the
>>code,
>> please just refer the package org.apache.airavata.gfac, do not refer
>>any of
>> the classes in org.apache.airavata.core.gfac.
>>
>> Best place to start with is referring is from test classes
>> (LocalProviderTest,GramProviderTest), from tehre you can start looking
>>in
>> to GFacAPI class and see how to execution is flawing.
>>
>> if you have further questions please post on the list and more than
>>happy
>> to help. I will be doing some documentation about the architecture,
>>once I
>> am done, will post in to the list. And we will be having an architecture
>> review this week, so please watch the mailing list, if possible please
>>try
>> to join us.
>>
>> Regards
>> Lahiru
>>
>> On Mon, Feb 18, 2013 at 1:25 PM, Sanjaya Medonsa <sanjayamrt@gmail.com
>> >wrote:
>>
>> > Thanks Suresh and Chris! It seems I am moving on the correct path. I
>>have
>> > followed the email thread on improved GFac architecture. Though I am
>>not
>> > entirely clear on the improved GFac architecture, proposed integration
>> with
>> > OODT is primarily based on the GFac extension, PreExecustionChain,
>>which
>> > has not been modified with the Architecture improvements (As per one
>>of
>> the
>> > replies from Lahiru, output extension is supported with new
>> Architecture. I
>> > assume input extension is also supported).
>> >
>> > I have looked into provenance manager and related implementation.
>>Still I
>> > am unclear how Airavata support provenance aware work flow processing.
>> >
>> > Best Regards,
>> > Sanjaya
>> >
>> > On Mon, Feb 18, 2013 at 6:35 AM, Suresh Marru <smarru@apache.org>
>>wrote:
>> >
>> > > Hi Sanjaya,
>> > >
>> > > This sounds very exciting. Both Airavata and OODT projects have good
>> > > synergies and have been long looking for volunteers who can bridge
>>them
>> > > both. Please do not hesitate to ask any questions to either or both
>>the
>> > dev
>> > > lists. The more engaged you are, you will find use cases and
>>feedback
>> > which
>> > > should help your MSc project.
>> > >
>> > > Your plan sounds good. If you are following dev list, you may have
>> > > noticed, the GFac architecture has been improved to properly support
>> this
>> > > kind of handler architecture.
>> > >
>> > > You may also want to look at Airavata Registry API which has
>> organically
>> > > emerged with the need, but can greatly benefit for OODT's provenance
>> > > capabilities.
>> > >
>> > > Suresh
>> > >
>> > > On Feb 17, 2013, at 1:25 PM, "Mattmann, Chris A (388J)" <
>> > > chris.a.mattmann@jpl.nasa.gov> wrote:
>> > >
>> > > > +1, sounds like a great idea Sanjaya!
>> > > >
>> > > > I'm copying dev@oodt so they can be in the conversation too.
>> > > >
>> > > > Cheers,
>> > > > Chris
>> > > >
>> > > > On 2/17/13 10:22 AM, "Sanjaya Medonsa" <sanjayamrt@gmail.com>
>>wrote:
>> > > >
>> > > >> Hi Dev Team,
>> > > >> As I have posted previously, I am working on a Apache Airavata +
>> > Apache
>> > > >> OODT integration as my MSc project. Following is one of the
>>possible
>> > > >> integration to leverage Apache OODT file management capability
>>into
>> > > Apache
>> > > >> Airavata. Please review the proposal and let me know your
>>feedback.
>> > > >>
>> > > >> Proposal To Use Apache OODT products as input to Apache Airavata
>> > > Workflows
>> > > >> and staging product files into node where execution happens
>> > > >>
>> > >
>> >
>>=20
>>=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>=3D
>> > > >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> > > >> 1. Introduce "Apache OODT Product" as a new GFacParameterType.
>>New
>> > > "Apache
>> > > >> OODT Product" input type can sepcify "Product ID" or "File Path
>>to
>> > > >> Product"
>> > > >> as an input to Apache Airavata workflows.
>> > > >> 2. Introduce new PreExecuteChain to retrieve Apache OODT Products
>> from
>> > > >> File
>> > > >> Manager Server managed using Apache OODT.
>> > > >> 1. Using Apache OODT File Manager componenet transfer Product
>>from
>> > > Server
>> > > >> to input directory of the application as configured using
>>XBaya-GUI
>> > > under
>> > > >> advanced configuration. (Here the assumption is that Products are
>> > > >> accesible
>> > > >> through Apache OODT File Manager server)
>> > > >> 2. Finally reset the input value to local file path. I think we
>>can
>> > > remove
>> > > >> the OODT Product parameter from invocation context and add new
>>file
>> > > >> parameter with value set to 'local path of the transferred
>> product'. I
>> > > am
>> > > >> not quite sure what are the implications of changing input
>>parameter
>> > > type
>> > > >> during the execution.
>> > > >>
>> > > >> Similar approach has been implemented for GridFTP and HTTP.
>> > > >>
>> > > >> Best Regards,
>> > > >> Sanjaya
>> > > >
>> > >
>> > >
>> >
>>
>>
>>
>> --
>> System Analyst Programmer
>> PTI Lab
>> Indiana University
>>