Mailing-List: contact dev-help@airavata.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airavata.apache.org
Received-SPF: pass (athena.apache.org: local policy)
Message-ID: <525E912B.4040002@iu.edu>
Date: Wed, 16 Oct 2013 09:14:19 -0400
From: Marlon Pierce <marpierc@iu.edu>
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6;
 rv:24.0) Gecko/20100101 Thunderbird/24.0.1
MIME-Version: 1.0
To: dev@airavata.apache.org
Subject: Re: Povenance Aware Workflow Processing
References: 
 <CAJEo2dEN5pZADSEC-JLCD2gzDGeQFF74bbYVjmxvF59iLN+e6Q@mail.gmail.com>
	<CE3D6566.FCF71%chris.mattmann@gmail.com>
	<CAJEo2dEp7fy5=Mkrz+hCxwdQL7Rn7d0=dOeo668TNREZh6q9Zg@mail.gmail.com>
	<371ED874-B7EE-4208-8EB9-CE8A39381091@apache.org>
	<CAJEo2dFndVa5S1O4x3MPn6GNZ085Nj1CsageJ+CzffQPHNgSuA@mail.gmail.com>
 <CAJEo2dGJia62mc5OcxXuFAjvt8E93htr8giNpxk2X2E+DXD8zw@mail.gmail.com>
In-Reply-To: 
 <CAJEo2dGJia62mc5OcxXuFAjvt8E93htr8giNpxk2X2E+DXD8zw@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hi Sanjaya--

Could you file a Jira ticket for this? We also welcome patches if you
want to contribute it.


Marlon

On 10/15/13 9:32 PM, Sanjaya Medonsa wrote:
> Hi Suresh,
>     I have further looked into the code to understand, how provenance aware
> workflow processing has been implemented in current code base. As you have
> stated, it seems functionality is broken. In WorkflowInterpreter class,
> there is a method called readProvenance (only place where ProvenanceReader
> is used) . It is a private method, no longer a part of the actual
> execution. I think, it's better to go back to version where provenance
> aware workflow processing working as expected to get better understand on
> the expected flow. Then fix the issues in current version. Could you please
> let me know, possible version that this particular functionality worked as
> expected ?
>
> Best Regards,
> Sanjaya
>
>
> On Tue, Oct 15, 2013 at 4:09 PM, Sanjaya Medonsa <sanjayamrt@gmail.com>wrote:
>
>> Thanks Suresh! I'll do further investigations and let the team know my
>> findings.
>>
>> Best Regards,
>> Sanjaya
>>
>>
>> On Tue, Oct 15, 2013 at 6:52 AM, Suresh Marru <smarru@apache.org> wrote:
>>
>>> Hi Sanjaya,
>>>
>>> XBaya and Workflow Interpreter went through some changes which probably
>>> dropped the condition which makes use of the enableProvenanceSmartRun flag.
>>> AFAIK, the feature did not get deprecated it should be one of those cases
>>> which was not actively tested during releases and fell through the cracks.
>>> Lets see if any one gets motivated to debug and fix this issue. And as
>>> always, we welcome your patches. Sorry about pointing to a unverified
>>> feature.
>>>
>>> Suresh
>>>
>>> On Oct 14, 2013, at 9:01 PM, Sanjaya Medonsa <sanjayamrt@gmail.com>
>>> wrote:
>>>
>>>> Hi Suresh,
>>>> I have gone through the research paper in item 6 and try to match it to
>>> the current Airavata implementation. I can see that Airavata collects
>>> provenance data and stored them in a registry. I couldn't able to find out
>>> a place where these data has been actually used to improve the workflow
>>> processing. Xbaya can be configured with option enableProvenanceSmartRun.
>>> It seems this particular configuration is no longer used. Since provenance
>>> aware data is already available in registry, is it the responsibility of
>>> the application to use these data to improve workflow processing based on
>>> provenance aware data ?
>>>> Thanks
>>>> Sanjaya
>>>>
>>>>
>>>> On Sat, Aug 24, 2013 at 7:47 AM, Chris Mattmann <mattmann@apache.org>
>>> wrote:
>>>> Hey Sanjaya,
>>>>
>>>> You may want to check out the Traceable Product product type policy
>>>> [1] as well as OODT-139 [2] which includes the PCS services that serve
>>>> up the information and provenance.
>>>>
>>>> Cheers,
>>>> Chris
>>>>
>>>> [1]
>>>>
>>> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/examp
>>>> les/trace/
>>>> [2] http://issues.apache.org/jira/browse/OODT-139
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Sanjaya Medonsa <sanjayamrt@gmail.com>
>>>> Reply-To: <dev@oodt.apache.org>
>>>> Date: Friday, August 23, 2013 6:30 PM
>>>> To: Airavata Dev <dev@airavata.apache.org>
>>>> Cc: "dev@oodt.apache.org" <dev@oodt.apache.org>
>>>> Subject: Re: Povenance Aware Workflow Processing
>>>>
>>>>> Thanks Suresh! I am interested in Provenance aware workflow processing
>>>>> where OODT integration could help greatly with rich metadata available.
>>>>> I'll  go through the item 6 and let you know my understanding.
>>>>>
>>>>> Sanjaya
>>>>>
>>>>>
>>>>> On Thu, Aug 8, 2013 at 7:55 PM, Suresh Marru <smarru@apache.org>
>>> wrote:
>>>>>> Hi Sanjaya, Lewis,
>>>>>>
>>>>>> Appreciate your interest on this topic, this is a very good question
>>> and
>>>>>> sorry for the delay in responding, these days I feel I am always in
>>>>>> catch
>>>>>> up mode.
>>>>>>
>>>>>> Let me separate this discussion into four parts: Provenance
>>> Gathering,
>>>>>> Provenance Analyzing, Provenance Reasoning, Provenance Aware
>>> executions.
>>>>>> Gathering: Airavata is instrumented with rich provenance centric
>>>>>> information model which captures each step and publishes to an event
>>>>>> bus.
>>>>>> This is encoded in a "workflow tracking" schema [1]. So the current
>>>>>> trunk
>>>>>> of Airavata publishes provenance rich information but not all is
>>>>>> guaranteed
>>>>>> to be captured in the registry.
>>>>>>
>>>>>> Analyzing: Significant work on provenance aspect related to Airavata
>>> is
>>>>>> a
>>>>>> by product of doctoral dissertation of Yogesh Simmhan, currently a
>>>>>> research
>>>>>> asst. professor at USC. His thesis [2] is a good source of reference
>>> on
>>>>>> this topic. Both the research and tools are in active development by
>>>>>> Prof.
>>>>>> Beth Plale's group at IU [3]. Theoretically speaking since both Karma
>>>>>> uses
>>>>>> the same workflow tracking schema and event bus, the tools should be
>>>>>> easily
>>>>>> integratable but did not have a motivating use case to do so.
>>>>>>
>>>>>> Reasoning: Phala is an extension to Airavata XBaya (its legacy LEAD
>>>>>> version) and Karma framework which provides context to the gathered
>>>>>> provenance. Couple of papers on this topic [4], [5].
>>>>>>
>>>>>> Executions: This is an active topic within Airavata. Current release
>>> of
>>>>>> Airavata supports Provenance aware workflow executions and is
>>> discussed
>>>>>> in
>>>>>> [6]. I think the OODT and Airavata integration will greatly benefit
>>> this
>>>>>> aspect. Airavata Registry metadata cataloging is very primitive. And
>>>>>> Rich
>>>>>> data management capabilities of OODT should nicely enhance airavata
>>>>>> provenance aware executions.
>>>>>>
>>>>>> Please let me know which of the above aspects you are interested in
>>> and
>>>>>> we
>>>>>> can discuss further.
>>>>>>
>>>>>> Suresh
>>>>>>
>>>>>> [1] -
>>>>>>
>>>>>>
>>> https://svn.apache.org/repos/asf/airavata/trunk/modules/commons/workflow-
>>>>>> tracking/src/main/resources/schemas/workflow_tracking_types.xsd
>>>>>> [2] - http://gradworks.umi.com/3297094.pdf
>>>>>> [3] - http://d2i.indiana.edu/provenance_karma
>>>>>> [4] -
>>>>>>
>>> http://link.springer.com/content/pdf/10.1007%2F978-3-540-85502-6_18.pdf
>>>>>> [5] -
>>>>>>
>>>>>>
>>> http://dl.acm.org/citation.cfm?id=1590963.1591575&coll=DL&dl=GUIDE&CFID=2
>>>>>> 38325269&CFTOKEN=73364358
>>>>>> [6] - http://dl.acm.org/citation.cfm?id=2359844
>>>>>>
>>>>>>
>>>>>> On Jul 22, 2013, at 10:57 AM, Lewis John Mcgibbney <
>>>>>> lewis.mcgibbney@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Sanjaya,
>>>>>>> I am seeing some traffic floating around RE: Airavata OODT
>>> integration
>>>>>> but I am not clued up.
>>>>>>> Can you point me to the code? The issues? The mailing list threads?
>>>>>>> I am also extremely interested in provenance tracking of
>>> (record???)
>>>>>> data which passes through OODT. I would like to expose this as some
>>>>>> stage
>>>>>> to people who access the data.
>>>>>>> Thank you so much.
>>>>>>>
>>>>>>> On Monday, July 22, 2013, Sanjaya Medonsa <sanjayamrt@gmail.com>
>>>>>> wrote:
>>>>>>>> Hi,
>>>>>>>> I am trying to find out ways to improve Airavata provenance aware
>>>>>> workflow
>>>>>>>> processing based on rich metadata available in Apache OODT. As
>>> part
>>>>>> of
>>>>>> this
>>>>>>>> task, I am investigating how Apache Airavata is currently
>>> performing
>>>>>>>> provenance aware workflow processing. I have looked into the
>>>>>> provenance
>>>>>>>> manager implementation, where I could see lot of data related to
>>>>>> workflow
>>>>>>>> execution is saved in registry. But I could see very little of
>>> use
>>>>>> of
>>>>>> those
>>>>>>>> data. Could some one please explain the details of provenance
>>> aware
>>>>>>>> processing in Airavata and any future plans on this area?
>>>>>>>>
>>>>>>>> Best Regards,
>>>>>>>> Sanjaya
>>>>>>>>
>>>>>>> --
>>>>>>> Lewis
>>>>>>>
>>>>>>
>>>>
>>>>
>>>