Mailing-List: contact dev-help@oodt.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@oodt.apache.org
Received-SPF: pass (athena.apache.org: domain of brbarkstrom@gmail.com
 designates 74.125.82.45 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAMz4=FaTDkbtJC=aXjVYR+2hhmEquUywZEJHwwpnP=KqtmxGjQ@mail.gmail.com>
References: 
 <CAJvLytry5CAO7eA=EJw62w_pg=4HWuLuSLpC2yjjj-Yg=XF4pA@mail.gmail.com>
	<D04060F0.17FA99%chris.a.mattmann@jpl.nasa.gov>
	<4B3A58EE-54EF-4FE9-A3DA-8EE876299430@apache.org>
	<CAMz4=FaTDkbtJC=aXjVYR+2hhmEquUywZEJHwwpnP=KqtmxGjQ@mail.gmail.com>
Date: Fri, 19 Sep 2014 17:40:35 -0400
Message-ID: 
 <CAOG8ptUDGKVDFOnWa+QSP=rbKMwvHziUffD26b2bKUmjiWQ5zg@mail.gmail.com>
Subject: Re: Evaluate Suitable Scientific Workflow Language for Airavata.
From: Bruce Barkstrom <brbarkstrom@gmail.com>
To: dev@oodt.apache.org, alekny7@gmail.com
Content-Type: multipart/alternative; boundary=047d7ba977fc859b45050371f35f

--047d7ba977fc859b45050371f35f
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Thanks very much for the documentation.

Just glancing through it, I'm not sure I'd describe production
by embedding a particular language in the basic description.
In the large-scale production systems I architected for NASA's
ERBE and CERES production, I started with a Data Flow Diagram.
Now, I'd probably move to UML before getting close to a language-
specific design.

A key part of our work was two years getting an agreement on
the naming convention of the files that tied in to the multiple
instruments, multiple products, and multiple versions of each
element in the provenance.  I did a formal description of those
structures in an ESI paper (I guess 2009) on configuration management
in large-scale satellite data production.  That paper contains
some example figures of the provenance graphs - which are
probably easiest to visualize if you get the pdf version and
view the graphs with a zoom of 2800% or so.

One point that was important is that our production flow was
not completely automated - and humans had to invoke the
proper connection between input files, various pieces of
production executables, and output files - including variations
from one month to the next a as well as match*ing *which satellite
was involved (plus some other configuration issues).  In the
long run, we also needed to build production schedules so
the archive could check quality control on the proper running
of the process.  That work required interaction between objects
and staff on the archive and the science teams, which makes
for the usual schedule complications that are an intrinsic
part of production planning on big projects.

Also, in some work I'm doing currently, I am quite concerned
about user search states (or production states) that come from
a Markov model that allows simulation of user loadings on
an archive.  I haven't gone into full detail on this yet - but it
may mean that the interaction between the user and the archive
needs to keep track of the state of both agents.  The basic
notion for this work is that the archive and the user are
playing a two agent cooperative game where the archive
knows what it contains and the user knows what his or her
target is.  I think that requires maintaining agent states across
multiple sessions.  I don't recall whether that violates the
original source of the RESTFUL design, so I'll need to go
back to the original to check.

I'll try to look at what you've sent in more detail and provide
some more comments.  Regardless, I appreciate the information.

Bruce B.


On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
alekny7@gmail.com> wrote:

> Hi,
>
> it is not dataflow instead focused on orchestrating REST services but you
> may find it useful datapoint - we created worfklow service that uses
> natively JavaScript and JSON to describe what happens during workflow
> execution:
> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>
> HTH,
>
> Alek
>
> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <smarru@apache.org> wrote:
>
> > Hi Chris,
> >
> > Great to hear OODT community will be interested in adopting a JSON base=
d
> > workflow language and potentially a web based composer as well. Airavat=
a
> > previously had BPEL support initially through a home grown implementati=
on
> > [1] by Alek Slominski and later through Apache ODE [2]. Also a white
> paper
> > [3] by Alek on this topic is an interesting read.
> >
> > I am of the same opinion that we should adopt something more modern as
> the
> > challenges from scientific workflows seems to be converging with the da=
ta
> > flow patterns in business workflows.
> >
> > It will be great if we can all compile a list of potential candidates a=
nd
> > hack them through.
> >
> > Suresh
> > [1] -
> > http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> > [2] -
> >
> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtim=
e_for_eScience_workflows
> > [3] -
> >
> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.p=
df
> >
> >
> > On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> > chris.a.mattmann@jpl.nasa.gov> wrote:
> >
> > > Hi Guys,
> > >
> > > I've been interested in this too - we don't per have a specific
> > > OODT workflow language, but we specific workflows using XML, and
> > > other configuration (we are also thinking of moving to JSON for
> > > this).
> > >
> > > In the past I've also looked at YAWL and BPEL - both seem complex
> > > to me.
> > >
> > > I wonder at the end of the day if we should adopt something more
> > > modern like PIG or some other data flow type of language (PIG
> > > is really neat).
> > >
> > > Cheers,
> > > Chris
> > >
> > >
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > Chris Mattmann, Ph.D.
> > > Chief Architect
> > > Instrument Software and Science Data Systems Section (398)
> > > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > > Office: 168-519, Mailstop: 168-527
> > > Email: chris.a.mattmann@nasa.gov
> > > WWW:  http://sunset.usc.edu/~mattmann/
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > Adjunct Associate Professor, Computer Science Department
> > > University of Southern California, Los Angeles, CA 90089 USA
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >
> > >
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Shameera Rathnayaka <shameerainfo@gmail.com>
> > > Reply-To: "architecture@airavata.apache.org"
> > > <architecture@airavata.apache.org>
> > > Date: Thursday, September 18, 2014 8:26 AM
> > > To: "architecture@airavata.apache.org" <
> architecture@airavata.apache.org
> > >,
> > > dev <dev@airavata.apache.org>
> > > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
> > >
> > >> Hi All,
> > >>
> > >> As we all know Airavata has its own workflow language call XWF. When
> XWF
> > >> was introduced, main focus points are interoperability and
> > convertibility.
> > >> But with years of experience it is convinced that above requirements
> are
> > >> not really useful when we come to real world use cases. And XWF is X=
ML
> > >> based bulky language where we attache WSDLs and Workflow image it
> self.
> > >> But
> > >> with the recent changes WSDL part is being removed from XWF.
> > >>
> > >> It is worth to evaluate handy Scientific workflow languages in
> industry
> > >> and
> > >> find out pros and cons, at the end of this evaluation we need to com=
e
> up
> > >> with idea how we should improve Airavata workflow language, either w=
e
> > can
> > >> improve existing XWF language, totally change to a new language
> > available
> > >> in industry or write a new light weight language. Basic requirements
> > that
> > >> we expect from new improvement are, high usability, flexible, light
> > weight
> > >> and real time monitoring support. As you can see above requirements
> are
> > >> not
> > >> direct comes with workflow languages but we need workflow language
> which
> > >> help to support above requirements.
> > >>
> > >> After reading few papers and googling, initially i have come up with
> > >> following three existing languages,
> > >> 1. YAWL <http://www.yawlfoundation.org/>
> > >> 2. WS-BPEL
> > >> =E2=80=8B3. SIDL
> > >> <http://computation.llnl.gov/casc/components/index.html#page=3Dhome>
> > >>
> > >> In my opinion SIDL is more familiar with scientific domain,
> Radical-SAGA
> > >> also uses slightly modified version of SIDL. Other than above three
> > >> languages we can come up with simple workflow language base on json(=
or
> > >> yaml) which support all our requirements for some extends.
> > >>
> > >> It would be grate if I can get more input regarding the $Subject for=
m
> > the
> > >> airavata community. You all are more than welcome to provide any typ=
e
> of
> > >> suggestions.
> > >>
> > >> Thanks,
> > >> Shameera.
> > >>
> > >> =E2=80=8B
> > >>
> > >> --
> > >> Best Regards,
> > >> Shameera Rathnayaka.
> > >>
> > >> email: shameera AT apache.org , shameerainfo AT gmail.com
> > >> Blog : http://shameerarathnayaka.blogspot.com/
> >
> >
>
>
> --
> The best way to predict the future is to invent it - Alan Kay
>

--047d7ba977fc859b45050371f35f--