Return-Path: X-Original-To: apmail-airavata-dev-archive@www.apache.org Delivered-To: apmail-airavata-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C9D97E17B for ; Wed, 20 Feb 2013 02:33:10 +0000 (UTC) Received: (qmail 83256 invoked by uid 500); 20 Feb 2013 02:33:10 -0000 Delivered-To: apmail-airavata-dev-archive@airavata.apache.org Received: (qmail 83158 invoked by uid 500); 20 Feb 2013 02:33:10 -0000 Mailing-List: contact dev-help@airavata.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airavata.apache.org Delivered-To: mailing list dev@airavata.apache.org Received: (qmail 83138 invoked by uid 99); 20 Feb 2013 02:33:09 -0000 Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Feb 2013 02:33:09 +0000 Received: from localhost (HELO [10.0.1.5]) (127.0.0.1) (smtp-auth username smarru, mechanism plain) by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Feb 2013 02:33:09 +0000 Subject: Re: Proposal To Use Apache OODT products as input to Apache Airavata Workflows and staging product files into node where execution happens Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Content-Type: text/plain; charset=iso-8859-1 From: Suresh Marru In-Reply-To: Date: Tue, 19 Feb 2013 21:33:09 -0500 Cc: "dev@oodt.apache.org" Content-Transfer-Encoding: quoted-printable Message-Id: References: <1B3C10EB-FFB0-46F0-B773-6A8D816C489E@apache.org> To: dev@airavata.apache.org X-Mailer: Apple Mail (2.1499) On Feb 19, 2013, at 1:52 PM, Sanjaya Medonsa = wrote: > Thanks Lahiru! I have gone through the test classes and the classes in > package org.apache.airavata.gfac. It was really helpful to understand = the > new architecture. I have listed down my approach based on new = architecture > to use Apache OODT products as an input to Airavata. > 1. Introduce new Data Type to represent Apache OODT Product as = an > DataType. Basically new DataType is added into the = GFacParameterTypes.xsd. > 2. With new Architecture In Handlers and Out Handlers replaces = the > Pre/Post execution chains in old architecture. For the moment I am = focusing > on using Apache OODT product ID or file path as an input and stage the = file > (product) into host where actual execution happens. File staging = requires > to retrieve product from a File Manager server to the host where = execution > occurs. File staging can be implemented as an* In Handler* and needs = to be > configured as a new item in the list of configured In Handlers. > 3. Handler should first verify the input parameter types = listed in > Service Description of the Application context of the = *JobExecutionContext*. > If input type matches the new parameter type, in handler stage file = into > host machine using Apache OODT file manager component. Corresponding = input > value can be retrieved from *In MessageContex*t. If a parameter type = in > MessageContext matches the new input type, then corresponding value is = the > id or file path to product managed by Apache OODT File Manager server. Hi Sanjaya, + 1 to this plan. You are right on!! great job in dissecting the code = and identifying where you need to plug in. Since you are bridging both = the projects, these kind of details will greatly help every one = understand what you are doing and offer early help.=20 Suresh > Best Regards, > Sanjaya >=20 > On Tue, Feb 19, 2013 at 1:29 AM, Lahiru Gunathilake = wrote: >=20 >> Hi Sanjaya, >>=20 >> If you want to understand the new architecture by looking in to the = code, >> please just refer the package org.apache.airavata.gfac, do not refer = any of >> the classes in org.apache.airavata.core.gfac. >>=20 >> Best place to start with is referring is from test classes >> (LocalProviderTest,GramProviderTest), from tehre you can start = looking in >> to GFacAPI class and see how to execution is flawing. >>=20 >> if you have further questions please post on the list and more than = happy >> to help. I will be doing some documentation about the architecture, = once I >> am done, will post in to the list. And we will be having an = architecture >> review this week, so please watch the mailing list, if possible = please try >> to join us. >>=20 >> Regards >> Lahiru >>=20 >> On Mon, Feb 18, 2013 at 1:25 PM, Sanjaya Medonsa = >> wrote: >>=20 >>> Thanks Suresh and Chris! It seems I am moving on the correct path. I = have >>> followed the email thread on improved GFac architecture. Though I am = not >>> entirely clear on the improved GFac architecture, proposed = integration >> with >>> OODT is primarily based on the GFac extension, PreExecustionChain, = which >>> has not been modified with the Architecture improvements (As per one = of >> the >>> replies from Lahiru, output extension is supported with new >> Architecture. I >>> assume input extension is also supported). >>>=20 >>> I have looked into provenance manager and related implementation. = Still I >>> am unclear how Airavata support provenance aware work flow = processing. >>>=20 >>> Best Regards, >>> Sanjaya >>>=20 >>> On Mon, Feb 18, 2013 at 6:35 AM, Suresh Marru = wrote: >>>=20 >>>> Hi Sanjaya, >>>>=20 >>>> This sounds very exciting. Both Airavata and OODT projects have = good >>>> synergies and have been long looking for volunteers who can bridge = them >>>> both. Please do not hesitate to ask any questions to either or both = the >>> dev >>>> lists. The more engaged you are, you will find use cases and = feedback >>> which >>>> should help your MSc project. >>>>=20 >>>> Your plan sounds good. If you are following dev list, you may have >>>> noticed, the GFac architecture has been improved to properly = support >> this >>>> kind of handler architecture. >>>>=20 >>>> You may also want to look at Airavata Registry API which has >> organically >>>> emerged with the need, but can greatly benefit for OODT's = provenance >>>> capabilities. >>>>=20 >>>> Suresh >>>>=20 >>>> On Feb 17, 2013, at 1:25 PM, "Mattmann, Chris A (388J)" < >>>> chris.a.mattmann@jpl.nasa.gov> wrote: >>>>=20 >>>>> +1, sounds like a great idea Sanjaya! >>>>>=20 >>>>> I'm copying dev@oodt so they can be in the conversation too. >>>>>=20 >>>>> Cheers, >>>>> Chris >>>>>=20 >>>>> On 2/17/13 10:22 AM, "Sanjaya Medonsa" = wrote: >>>>>=20 >>>>>> Hi Dev Team, >>>>>> As I have posted previously, I am working on a Apache Airavata + >>> Apache >>>>>> OODT integration as my MSc project. Following is one of the = possible >>>>>> integration to leverage Apache OODT file management capability = into >>>> Apache >>>>>> Airavata. Please review the proposal and let me know your = feedback. >>>>>>=20 >>>>>> Proposal To Use Apache OODT products as input to Apache Airavata >>>> Workflows >>>>>> and staging product files into node where execution happens >>>>>>=20 >>>>=20 >>>=20 >> = =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D >>>>>> 1. Introduce "Apache OODT Product" as a new GFacParameterType. = New >>>> "Apache >>>>>> OODT Product" input type can sepcify "Product ID" or "File Path = to >>>>>> Product" >>>>>> as an input to Apache Airavata workflows. >>>>>> 2. Introduce new PreExecuteChain to retrieve Apache OODT Products >> from >>>>>> File >>>>>> Manager Server managed using Apache OODT. >>>>>> 1. Using Apache OODT File Manager componenet transfer Product = from >>>> Server >>>>>> to input directory of the application as configured using = XBaya-GUI >>>> under >>>>>> advanced configuration. (Here the assumption is that Products are >>>>>> accesible >>>>>> through Apache OODT File Manager server) >>>>>> 2. Finally reset the input value to local file path. I think we = can >>>> remove >>>>>> the OODT Product parameter from invocation context and add new = file >>>>>> parameter with value set to 'local path of the transferred >> product'. I >>>> am >>>>>> not quite sure what are the implications of changing input = parameter >>>> type >>>>>> during the execution. >>>>>>=20 >>>>>> Similar approach has been implemented for GridFTP and HTTP. >>>>>>=20 >>>>>> Best Regards, >>>>>> Sanjaya >>>>>=20 >>>>=20 >>>>=20 >>>=20 >>=20 >>=20 >>=20 >> -- >> System Analyst Programmer >> PTI Lab >> Indiana University >>=20