airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marlon Pierce <marpi...@iu.edu>
Subject Re: Povenance Aware Workflow Processing
Date Wed, 16 Oct 2013 13:14:19 GMT
Hi Sanjaya--

Could you file a Jira ticket for this? We also welcome patches if you
want to contribute it.


Marlon

On 10/15/13 9:32 PM, Sanjaya Medonsa wrote:
> Hi Suresh,
>     I have further looked into the code to understand, how provenance aware
> workflow processing has been implemented in current code base. As you have
> stated, it seems functionality is broken. In WorkflowInterpreter class,
> there is a method called readProvenance (only place where ProvenanceReader
> is used) . It is a private method, no longer a part of the actual
> execution. I think, it's better to go back to version where provenance
> aware workflow processing working as expected to get better understand on
> the expected flow. Then fix the issues in current version. Could you please
> let me know, possible version that this particular functionality worked as
> expected ?
>
> Best Regards,
> Sanjaya
>
>
> On Tue, Oct 15, 2013 at 4:09 PM, Sanjaya Medonsa <sanjayamrt@gmail.com>wrote:
>
>> Thanks Suresh! I'll do further investigations and let the team know my
>> findings.
>>
>> Best Regards,
>> Sanjaya
>>
>>
>> On Tue, Oct 15, 2013 at 6:52 AM, Suresh Marru <smarru@apache.org> wrote:
>>
>>> Hi Sanjaya,
>>>
>>> XBaya and Workflow Interpreter went through some changes which probably
>>> dropped the condition which makes use of the enableProvenanceSmartRun flag.
>>> AFAIK, the feature did not get deprecated it should be one of those cases
>>> which was not actively tested during releases and fell through the cracks.
>>> Lets see if any one gets motivated to debug and fix this issue. And as
>>> always, we welcome your patches. Sorry about pointing to a unverified
>>> feature.
>>>
>>> Suresh
>>>
>>> On Oct 14, 2013, at 9:01 PM, Sanjaya Medonsa <sanjayamrt@gmail.com>
>>> wrote:
>>>
>>>> Hi Suresh,
>>>> I have gone through the research paper in item 6 and try to match it to
>>> the current Airavata implementation. I can see that Airavata collects
>>> provenance data and stored them in a registry. I couldn't able to find out
>>> a place where these data has been actually used to improve the workflow
>>> processing. Xbaya can be configured with option enableProvenanceSmartRun.
>>> It seems this particular configuration is no longer used. Since provenance
>>> aware data is already available in registry, is it the responsibility of
>>> the application to use these data to improve workflow processing based on
>>> provenance aware data ?
>>>> Thanks
>>>> Sanjaya
>>>>
>>>>
>>>> On Sat, Aug 24, 2013 at 7:47 AM, Chris Mattmann <mattmann@apache.org>
>>> wrote:
>>>> Hey Sanjaya,
>>>>
>>>> You may want to check out the Traceable Product product type policy
>>>> [1] as well as OODT-139 [2] which includes the PCS services that serve
>>>> up the information and provenance.
>>>>
>>>> Cheers,
>>>> Chris
>>>>
>>>> [1]
>>>>
>>> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/examp
>>>> les/trace/
>>>> [2] http://issues.apache.org/jira/browse/OODT-139
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Sanjaya Medonsa <sanjayamrt@gmail.com>
>>>> Reply-To: <dev@oodt.apache.org>
>>>> Date: Friday, August 23, 2013 6:30 PM
>>>> To: Airavata Dev <dev@airavata.apache.org>
>>>> Cc: "dev@oodt.apache.org" <dev@oodt.apache.org>
>>>> Subject: Re: Povenance Aware Workflow Processing
>>>>
>>>>> Thanks Suresh! I am interested in Provenance aware workflow processing
>>>>> where OODT integration could help greatly with rich metadata available.
>>>>> I'll  go through the item 6 and let you know my understanding.
>>>>>
>>>>> Sanjaya
>>>>>
>>>>>
>>>>> On Thu, Aug 8, 2013 at 7:55 PM, Suresh Marru <smarru@apache.org>
>>> wrote:
>>>>>> Hi Sanjaya, Lewis,
>>>>>>
>>>>>> Appreciate your interest on this topic, this is a very good question
>>> and
>>>>>> sorry for the delay in responding, these days I feel I am always
in
>>>>>> catch
>>>>>> up mode.
>>>>>>
>>>>>> Let me separate this discussion into four parts: Provenance
>>> Gathering,
>>>>>> Provenance Analyzing, Provenance Reasoning, Provenance Aware
>>> executions.
>>>>>> Gathering: Airavata is instrumented with rich provenance centric
>>>>>> information model which captures each step and publishes to an event
>>>>>> bus.
>>>>>> This is encoded in a "workflow tracking" schema [1]. So the current
>>>>>> trunk
>>>>>> of Airavata publishes provenance rich information but not all is
>>>>>> guaranteed
>>>>>> to be captured in the registry.
>>>>>>
>>>>>> Analyzing: Significant work on provenance aspect related to Airavata
>>> is
>>>>>> a
>>>>>> by product of doctoral dissertation of Yogesh Simmhan, currently
a
>>>>>> research
>>>>>> asst. professor at USC. His thesis [2] is a good source of reference
>>> on
>>>>>> this topic. Both the research and tools are in active development
by
>>>>>> Prof.
>>>>>> Beth Plale's group at IU [3]. Theoretically speaking since both Karma
>>>>>> uses
>>>>>> the same workflow tracking schema and event bus, the tools should
be
>>>>>> easily
>>>>>> integratable but did not have a motivating use case to do so.
>>>>>>
>>>>>> Reasoning: Phala is an extension to Airavata XBaya (its legacy LEAD
>>>>>> version) and Karma framework which provides context to the gathered
>>>>>> provenance. Couple of papers on this topic [4], [5].
>>>>>>
>>>>>> Executions: This is an active topic within Airavata. Current release
>>> of
>>>>>> Airavata supports Provenance aware workflow executions and is
>>> discussed
>>>>>> in
>>>>>> [6]. I think the OODT and Airavata integration will greatly benefit
>>> this
>>>>>> aspect. Airavata Registry metadata cataloging is very primitive.
And
>>>>>> Rich
>>>>>> data management capabilities of OODT should nicely enhance airavata
>>>>>> provenance aware executions.
>>>>>>
>>>>>> Please let me know which of the above aspects you are interested
in
>>> and
>>>>>> we
>>>>>> can discuss further.
>>>>>>
>>>>>> Suresh
>>>>>>
>>>>>> [1] -
>>>>>>
>>>>>>
>>> https://svn.apache.org/repos/asf/airavata/trunk/modules/commons/workflow-
>>>>>> tracking/src/main/resources/schemas/workflow_tracking_types.xsd
>>>>>> [2] - http://gradworks.umi.com/3297094.pdf
>>>>>> [3] - http://d2i.indiana.edu/provenance_karma
>>>>>> [4] -
>>>>>>
>>> http://link.springer.com/content/pdf/10.1007%2F978-3-540-85502-6_18.pdf
>>>>>> [5] -
>>>>>>
>>>>>>
>>> http://dl.acm.org/citation.cfm?id=1590963.1591575&coll=DL&dl=GUIDE&CFID=2
>>>>>> 38325269&CFTOKEN=73364358
>>>>>> [6] - http://dl.acm.org/citation.cfm?id=2359844
>>>>>>
>>>>>>
>>>>>> On Jul 22, 2013, at 10:57 AM, Lewis John Mcgibbney <
>>>>>> lewis.mcgibbney@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Sanjaya,
>>>>>>> I am seeing some traffic floating around RE: Airavata OODT
>>> integration
>>>>>> but I am not clued up.
>>>>>>> Can you point me to the code? The issues? The mailing list threads?
>>>>>>> I am also extremely interested in provenance tracking of
>>> (record???)
>>>>>> data which passes through OODT. I would like to expose this as some
>>>>>> stage
>>>>>> to people who access the data.
>>>>>>> Thank you so much.
>>>>>>>
>>>>>>> On Monday, July 22, 2013, Sanjaya Medonsa <sanjayamrt@gmail.com>
>>>>>> wrote:
>>>>>>>> Hi,
>>>>>>>> I am trying to find out ways to improve Airavata provenance
aware
>>>>>> workflow
>>>>>>>> processing based on rich metadata available in Apache OODT.
As
>>> part
>>>>>> of
>>>>>> this
>>>>>>>> task, I am investigating how Apache Airavata is currently
>>> performing
>>>>>>>> provenance aware workflow processing. I have looked into
the
>>>>>> provenance
>>>>>>>> manager implementation, where I could see lot of data related
to
>>>>>> workflow
>>>>>>>> execution is saved in registry. But I could see very little
of
>>> use
>>>>>> of
>>>>>> those
>>>>>>>> data. Could some one please explain the details of provenance
>>> aware
>>>>>>>> processing in Airavata and any future plans on this area?
>>>>>>>>
>>>>>>>> Best Regards,
>>>>>>>> Sanjaya
>>>>>>>>
>>>>>>> --
>>>>>>> Lewis
>>>>>>>
>>>>>>
>>>>
>>>>
>>>


Mime
View raw message