airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yasas Gunarathne <>
Subject Re: User Defined Workflow Execution Framework
Date Sun, 29 Apr 2018 05:57:40 GMT
Hi All,

Thank you very much for the information. I did a research on the internals
of orchestrator and the new helix based workflow (lower level) execution,
throughout past few weeks.

Even though helix supports adding any number of experiments (i.e. complete
experiments including pre and post workflows) chained together, it is
required to maintain a higher level workflow manager as a separate layer
for orchestrator and submit experiments one after the other (if cannot be
run parallely) or parallelly (if execution is independent)  to preserve the
fault tolerance and enable flow handling of the higher level workflow.

Therefore, the steps that the new layer of orchestrator is supposed to do
will be,

   1. Parsing the provided high level workflow schema and arrange the list
   of experiments.
   2. Sending experiments according to the provided order and save their
   results in the storage resource.
   3. If there are dependencies (Result of an experiment is required to
   generate the input for another experiment), managing them accordingly while
   providing support to do modifications to the results in between.
   4. Providing flow handling methods (Start, Stop, Pause, Resume, Restart)

I have attached few simple top level workflow examples to support the
explanation. Please provide your valuable suggestions.


On Mon, Mar 26, 2018 at 8:43 AM, Suresh Marru <> wrote:

> Hi Yasas,
> Dimuthu already clarified but let me add few more points.
> Thats a very good questions, interpreter vs compiler (in the context of
> workflows). Yes Airavata historically took the interpreter approach, where
> after execution of each node in a workflows, the execution comes back to
> the enactment engine and re-inspects the state. This facilitated user
> interactively through executions. Attached state transition diagram might
> illustrate it more.
> Back to the current scope, I think you got the over all goal correct and
> your approach is reasonable. There are some details which are missing but
> thats expected. Just be aware that if your project is accepted, you will
> need to work with airavata community over the summer and refine the
> implementation details as you go. You are on a good start.
> Cheers,
> Suresh
> On Mar 25, 2018, at 8:44 PM, DImuthu Upeksha <>
> wrote:
> Hi Yasas,
> I'm not a expert in XBaya design and use cases but I think Suresh can shed
> some light about it. However we no longer use XBaya for workflow
> interpretation. So don't get confused with the workflows defined in XBaya
> with the description provided in the JIRA ticket. Let's try to make the
> concepts clear. We need two levels of Workflows.
> 1. To run a single experiment of an Application. We call this as a DAG. So
> a DAG is statically defined. It can have a set of environment setup tasks,
> data staging tasks and a job submission task. For example, a DAG is created
> to run a  Gaussian experiment on a compute host
> 2. To make a chain of Applications. This is what we call an actual
> Workflow. In a workflow you can have a Gaussian Experiment running and
> followed by a Lammps Experiment. So this is a dynamic workflow. Users can
> come up with different combinations of Applications as a workflow
> However your claim is true about pausing and restarting workflows. Either
> it is a statically defined DAG or a dynamic workflow, we should be able to
> do those operations.
> I can understand some of the words and terminologies that are in those
> resources are confusing and unclear so please feel free to let us know if
> you need anything to be clarified.
> Thanks
> Dimuthu
> On Sun, Mar 25, 2018 at 2:45 AM, Yasas Gunarathne <
>> wrote:
>> Hi All,
>> I have few questions to be clarified regarding the user-defined workflow
>> execution in Apache Airavata. Here I am talking about the high level
>> workflows that are used to chain together multiple applications. This
>> related to the issue - Airavata-2717 [1].
>> In this [2] documentation it says that, the workflow interpreter that
>> worked with XBaya provided an interpreted workflow execution framework
>> rather than the compiled workflow execution environments, which allowed the
>> users to pause the execution of the workflow as necessary and update the
>> DAG’s execution states or even the DAG itself and resume execution.
>> I want to know the actual requirement of having an interpreted workflow
>> execution at this level. Is there any domain level advantage in allowing
>> users to modify the order of workflow at runtime?
>> I think we can have, pause, resume, restart, and stop commands available
>> even in a compiled workflow execution environment, as long as we don't need
>> to change the workflow.
>> [1]
>> [2]
>> Regards
>> --
>> *Yasas Gunarathne*
>> Undergraduate at Department of Computer Science and Engineering
>> Faculty of Engineering - University of Moratuwa Sri Lanka
>> LinkedIn <> | GitHub
>> <> | Mobile : +94 77 4893616
>> <+94%2077%20489%203616>

*Yasas Gunarathne*
Undergraduate at Department of Computer Science and Engineering
Faculty of Engineering - University of Moratuwa Sri Lanka
LinkedIn <> | GitHub
<> | Mobile : +94 77 4893616

View raw message