airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sanjaya Medonsa <sanjaya...@gmail.com>
Subject Re: Povenance Aware Workflow Processing
Date Tue, 15 Oct 2013 10:39:56 GMT
Thanks Suresh! I'll do further investigations and let the team know my
findings.

Best Regards,
Sanjaya


On Tue, Oct 15, 2013 at 6:52 AM, Suresh Marru <smarru@apache.org> wrote:

> Hi Sanjaya,
>
> XBaya and Workflow Interpreter went through some changes which probably
> dropped the condition which makes use of the enableProvenanceSmartRun flag.
> AFAIK, the feature did not get deprecated it should be one of those cases
> which was not actively tested during releases and fell through the cracks.
> Lets see if any one gets motivated to debug and fix this issue. And as
> always, we welcome your patches. Sorry about pointing to a unverified
> feature.
>
> Suresh
>
> On Oct 14, 2013, at 9:01 PM, Sanjaya Medonsa <sanjayamrt@gmail.com> wrote:
>
> > Hi Suresh,
> > I have gone through the research paper in item 6 and try to match it to
> the current Airavata implementation. I can see that Airavata collects
> provenance data and stored them in a registry. I couldn't able to find out
> a place where these data has been actually used to improve the workflow
> processing. Xbaya can be configured with option enableProvenanceSmartRun.
> It seems this particular configuration is no longer used. Since provenance
> aware data is already available in registry, is it the responsibility of
> the application to use these data to improve workflow processing based on
> provenance aware data ?
> >
> > Thanks
> > Sanjaya
> >
> >
> > On Sat, Aug 24, 2013 at 7:47 AM, Chris Mattmann <mattmann@apache.org>
> wrote:
> > Hey Sanjaya,
> >
> > You may want to check out the Traceable Product product type policy
> > [1] as well as OODT-139 [2] which includes the PCS services that serve
> > up the information and provenance.
> >
> > Cheers,
> > Chris
> >
> > [1]
> >
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/examp
> > les/trace/
> > [2] http://issues.apache.org/jira/browse/OODT-139
> >
> >
> > -----Original Message-----
> > From: Sanjaya Medonsa <sanjayamrt@gmail.com>
> > Reply-To: <dev@oodt.apache.org>
> > Date: Friday, August 23, 2013 6:30 PM
> > To: Airavata Dev <dev@airavata.apache.org>
> > Cc: "dev@oodt.apache.org" <dev@oodt.apache.org>
> > Subject: Re: Povenance Aware Workflow Processing
> >
> > >Thanks Suresh! I am interested in Provenance aware workflow processing
> > >where OODT integration could help greatly with rich metadata available.
> > >I'll  go through the item 6 and let you know my understanding.
> > >
> > >Sanjaya
> > >
> > >
> > >On Thu, Aug 8, 2013 at 7:55 PM, Suresh Marru <smarru@apache.org> wrote:
> > >
> > >> Hi Sanjaya, Lewis,
> > >>
> > >> Appreciate your interest on this topic, this is a very good question
> and
> > >> sorry for the delay in responding, these days I feel I am always in
> > >>catch
> > >> up mode.
> > >>
> > >> Let me separate this discussion into four parts: Provenance Gathering,
> > >> Provenance Analyzing, Provenance Reasoning, Provenance Aware
> executions.
> > >>
> > >> Gathering: Airavata is instrumented with rich provenance centric
> > >> information model which captures each step and publishes to an event
> > >>bus.
> > >> This is encoded in a "workflow tracking" schema [1]. So the current
> > >>trunk
> > >> of Airavata publishes provenance rich information but not all is
> > >>guaranteed
> > >> to be captured in the registry.
> > >>
> > >> Analyzing: Significant work on provenance aspect related to Airavata
> is
> > >>a
> > >> by product of doctoral dissertation of Yogesh Simmhan, currently a
> > >>research
> > >> asst. professor at USC. His thesis [2] is a good source of reference
> on
> > >> this topic. Both the research and tools are in active development by
> > >>Prof.
> > >> Beth Plale's group at IU [3]. Theoretically speaking since both Karma
> > >>uses
> > >> the same workflow tracking schema and event bus, the tools should be
> > >>easily
> > >> integratable but did not have a motivating use case to do so.
> > >>
> > >> Reasoning: Phala is an extension to Airavata XBaya (its legacy LEAD
> > >> version) and Karma framework which provides context to the gathered
> > >> provenance. Couple of papers on this topic [4], [5].
> > >>
> > >> Executions: This is an active topic within Airavata. Current release
> of
> > >> Airavata supports Provenance aware workflow executions and is
> discussed
> > >>in
> > >> [6]. I think the OODT and Airavata integration will greatly benefit
> this
> > >> aspect. Airavata Registry metadata cataloging is very primitive. And
> > >>Rich
> > >> data management capabilities of OODT should nicely enhance airavata
> > >> provenance aware executions.
> > >>
> > >> Please let me know which of the above aspects you are interested in
> and
> > >>we
> > >> can discuss further.
> > >>
> > >> Suresh
> > >>
> > >> [1] -
> > >>
> > >>
> https://svn.apache.org/repos/asf/airavata/trunk/modules/commons/workflow-
> > >>tracking/src/main/resources/schemas/workflow_tracking_types.xsd
> > >> [2] - http://gradworks.umi.com/3297094.pdf
> > >> [3] - http://d2i.indiana.edu/provenance_karma
> > >> [4] -
> > >>
> http://link.springer.com/content/pdf/10.1007%2F978-3-540-85502-6_18.pdf
> > >> [5] -
> > >>
> > >>
> http://dl.acm.org/citation.cfm?id=1590963.1591575&coll=DL&dl=GUIDE&CFID=2
> > >>38325269&CFTOKEN=73364358
> > >> [6] - http://dl.acm.org/citation.cfm?id=2359844
> > >>
> > >>
> > >> On Jul 22, 2013, at 10:57 AM, Lewis John Mcgibbney <
> > >> lewis.mcgibbney@gmail.com> wrote:
> > >>
> > >> > Hi Sanjaya,
> > >> > I am seeing some traffic floating around RE: Airavata OODT
> integration
> > >> but I am not clued up.
> > >> > Can you point me to the code? The issues? The mailing list threads?
> > >> > I am also extremely interested in provenance tracking of (record???)
> > >> data which passes through OODT. I would like to expose this as some
> > >>stage
> > >> to people who access the data.
> > >> > Thank you so much.
> > >> >
> > >> > On Monday, July 22, 2013, Sanjaya Medonsa <sanjayamrt@gmail.com>
> > >>wrote:
> > >> > > Hi,
> > >> > > I am trying to find out ways to improve Airavata provenance aware
> > >> workflow
> > >> > > processing based on rich metadata available in Apache OODT. As
> part
> > >>of
> > >> this
> > >> > > task, I am investigating how Apache Airavata is currently
> performing
> > >> > > provenance aware workflow processing. I have looked into the
> > >>provenance
> > >> > > manager implementation, where I could see lot of data related
to
> > >> workflow
> > >> > > execution is saved in registry. But I could see very little of
use
> > >>of
> > >> those
> > >> > > data. Could some one please explain the details of provenance
> aware
> > >> > > processing in Airavata and any future plans on this area?
> > >> > >
> > >> > > Best Regards,
> > >> > > Sanjaya
> > >> > >
> > >> >
> > >> > --
> > >> > Lewis
> > >> >
> > >>
> > >>
> >
> >
> >
>
>

Mime
View raw message