airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sanjaya Medonsa <sanjaya...@gmail.com>
Subject Re: Povenance Aware Workflow Processing
Date Wed, 16 Oct 2013 01:32:15 GMT
Hi Suresh,
    I have further looked into the code to understand, how provenance aware
workflow processing has been implemented in current code base. As you have
stated, it seems functionality is broken. In WorkflowInterpreter class,
there is a method called readProvenance (only place where ProvenanceReader
is used) . It is a private method, no longer a part of the actual
execution. I think, it's better to go back to version where provenance
aware workflow processing working as expected to get better understand on
the expected flow. Then fix the issues in current version. Could you please
let me know, possible version that this particular functionality worked as
expected ?

Best Regards,
Sanjaya


On Tue, Oct 15, 2013 at 4:09 PM, Sanjaya Medonsa <sanjayamrt@gmail.com>wrote:

> Thanks Suresh! I'll do further investigations and let the team know my
> findings.
>
> Best Regards,
> Sanjaya
>
>
> On Tue, Oct 15, 2013 at 6:52 AM, Suresh Marru <smarru@apache.org> wrote:
>
>> Hi Sanjaya,
>>
>> XBaya and Workflow Interpreter went through some changes which probably
>> dropped the condition which makes use of the enableProvenanceSmartRun flag.
>> AFAIK, the feature did not get deprecated it should be one of those cases
>> which was not actively tested during releases and fell through the cracks.
>> Lets see if any one gets motivated to debug and fix this issue. And as
>> always, we welcome your patches. Sorry about pointing to a unverified
>> feature.
>>
>> Suresh
>>
>> On Oct 14, 2013, at 9:01 PM, Sanjaya Medonsa <sanjayamrt@gmail.com>
>> wrote:
>>
>> > Hi Suresh,
>> > I have gone through the research paper in item 6 and try to match it to
>> the current Airavata implementation. I can see that Airavata collects
>> provenance data and stored them in a registry. I couldn't able to find out
>> a place where these data has been actually used to improve the workflow
>> processing. Xbaya can be configured with option enableProvenanceSmartRun.
>> It seems this particular configuration is no longer used. Since provenance
>> aware data is already available in registry, is it the responsibility of
>> the application to use these data to improve workflow processing based on
>> provenance aware data ?
>> >
>> > Thanks
>> > Sanjaya
>> >
>> >
>> > On Sat, Aug 24, 2013 at 7:47 AM, Chris Mattmann <mattmann@apache.org>
>> wrote:
>> > Hey Sanjaya,
>> >
>> > You may want to check out the Traceable Product product type policy
>> > [1] as well as OODT-139 [2] which includes the PCS services that serve
>> > up the information and provenance.
>> >
>> > Cheers,
>> > Chris
>> >
>> > [1]
>> >
>> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/examp
>> > les/trace/
>> > [2] http://issues.apache.org/jira/browse/OODT-139
>> >
>> >
>> > -----Original Message-----
>> > From: Sanjaya Medonsa <sanjayamrt@gmail.com>
>> > Reply-To: <dev@oodt.apache.org>
>> > Date: Friday, August 23, 2013 6:30 PM
>> > To: Airavata Dev <dev@airavata.apache.org>
>> > Cc: "dev@oodt.apache.org" <dev@oodt.apache.org>
>> > Subject: Re: Povenance Aware Workflow Processing
>> >
>> > >Thanks Suresh! I am interested in Provenance aware workflow processing
>> > >where OODT integration could help greatly with rich metadata available.
>> > >I'll  go through the item 6 and let you know my understanding.
>> > >
>> > >Sanjaya
>> > >
>> > >
>> > >On Thu, Aug 8, 2013 at 7:55 PM, Suresh Marru <smarru@apache.org>
>> wrote:
>> > >
>> > >> Hi Sanjaya, Lewis,
>> > >>
>> > >> Appreciate your interest on this topic, this is a very good question
>> and
>> > >> sorry for the delay in responding, these days I feel I am always in
>> > >>catch
>> > >> up mode.
>> > >>
>> > >> Let me separate this discussion into four parts: Provenance
>> Gathering,
>> > >> Provenance Analyzing, Provenance Reasoning, Provenance Aware
>> executions.
>> > >>
>> > >> Gathering: Airavata is instrumented with rich provenance centric
>> > >> information model which captures each step and publishes to an event
>> > >>bus.
>> > >> This is encoded in a "workflow tracking" schema [1]. So the current
>> > >>trunk
>> > >> of Airavata publishes provenance rich information but not all is
>> > >>guaranteed
>> > >> to be captured in the registry.
>> > >>
>> > >> Analyzing: Significant work on provenance aspect related to Airavata
>> is
>> > >>a
>> > >> by product of doctoral dissertation of Yogesh Simmhan, currently a
>> > >>research
>> > >> asst. professor at USC. His thesis [2] is a good source of reference
>> on
>> > >> this topic. Both the research and tools are in active development by
>> > >>Prof.
>> > >> Beth Plale's group at IU [3]. Theoretically speaking since both Karma
>> > >>uses
>> > >> the same workflow tracking schema and event bus, the tools should be
>> > >>easily
>> > >> integratable but did not have a motivating use case to do so.
>> > >>
>> > >> Reasoning: Phala is an extension to Airavata XBaya (its legacy LEAD
>> > >> version) and Karma framework which provides context to the gathered
>> > >> provenance. Couple of papers on this topic [4], [5].
>> > >>
>> > >> Executions: This is an active topic within Airavata. Current release
>> of
>> > >> Airavata supports Provenance aware workflow executions and is
>> discussed
>> > >>in
>> > >> [6]. I think the OODT and Airavata integration will greatly benefit
>> this
>> > >> aspect. Airavata Registry metadata cataloging is very primitive. And
>> > >>Rich
>> > >> data management capabilities of OODT should nicely enhance airavata
>> > >> provenance aware executions.
>> > >>
>> > >> Please let me know which of the above aspects you are interested in
>> and
>> > >>we
>> > >> can discuss further.
>> > >>
>> > >> Suresh
>> > >>
>> > >> [1] -
>> > >>
>> > >>
>> https://svn.apache.org/repos/asf/airavata/trunk/modules/commons/workflow-
>> > >>tracking/src/main/resources/schemas/workflow_tracking_types.xsd
>> > >> [2] - http://gradworks.umi.com/3297094.pdf
>> > >> [3] - http://d2i.indiana.edu/provenance_karma
>> > >> [4] -
>> > >>
>> http://link.springer.com/content/pdf/10.1007%2F978-3-540-85502-6_18.pdf
>> > >> [5] -
>> > >>
>> > >>
>> http://dl.acm.org/citation.cfm?id=1590963.1591575&coll=DL&dl=GUIDE&CFID=2
>> > >>38325269&CFTOKEN=73364358
>> > >> [6] - http://dl.acm.org/citation.cfm?id=2359844
>> > >>
>> > >>
>> > >> On Jul 22, 2013, at 10:57 AM, Lewis John Mcgibbney <
>> > >> lewis.mcgibbney@gmail.com> wrote:
>> > >>
>> > >> > Hi Sanjaya,
>> > >> > I am seeing some traffic floating around RE: Airavata OODT
>> integration
>> > >> but I am not clued up.
>> > >> > Can you point me to the code? The issues? The mailing list threads?
>> > >> > I am also extremely interested in provenance tracking of
>> (record???)
>> > >> data which passes through OODT. I would like to expose this as some
>> > >>stage
>> > >> to people who access the data.
>> > >> > Thank you so much.
>> > >> >
>> > >> > On Monday, July 22, 2013, Sanjaya Medonsa <sanjayamrt@gmail.com>
>> > >>wrote:
>> > >> > > Hi,
>> > >> > > I am trying to find out ways to improve Airavata provenance
aware
>> > >> workflow
>> > >> > > processing based on rich metadata available in Apache OODT.
As
>> part
>> > >>of
>> > >> this
>> > >> > > task, I am investigating how Apache Airavata is currently
>> performing
>> > >> > > provenance aware workflow processing. I have looked into
the
>> > >>provenance
>> > >> > > manager implementation, where I could see lot of data related
to
>> > >> workflow
>> > >> > > execution is saved in registry. But I could see very little
of
>> use
>> > >>of
>> > >> those
>> > >> > > data. Could some one please explain the details of provenance
>> aware
>> > >> > > processing in Airavata and any future plans on this area?
>> > >> > >
>> > >> > > Best Regards,
>> > >> > > Sanjaya
>> > >> > >
>> > >> >
>> > >> > --
>> > >> > Lewis
>> > >> >
>> > >>
>> > >>
>> >
>> >
>> >
>>
>>
>

Mime
View raw message