Hi All,

Thank you very much for the information. I did a research on the internals of orchestrator and the new helix based workflow (lower level) execution, throughout past few weeks.

Even though helix supports adding any number of experiments (i.e. complete experiments including pre and post workflows) chained together, it is required to maintain a higher level workflow manager as a separate layer for orchestrator and submit experiments one after the other (if cannot be run parallely) or parallelly (if execution is independent)  to preserve the fault tolerance and enable flow handling of the higher level workflow.

Therefore, the steps that the new layer of orchestrator is supposed to do will be,
  1. Parsing the provided high level workflow schema and arrange the list of experiments.
  2. Sending experiments according to the provided order and save their results in the storage resource.
  3. If there are dependencies (Result of an experiment is required to generate the input for another experiment), managing them accordingly while providing support to do modifications to the results in between.
  4. Providing flow handling methods (Start, Stop, Pause, Resume, Restart)
I have attached few simple top level workflow examples to support the explanation. Please provide your valuable suggestions.

Regards


On Mon, Mar 26, 2018 at 8:43 AM, Suresh Marru <smarru@apache.org> wrote:
Hi Yasas,

Dimuthu already clarified but let me add few more points. 

Thats a very good questions, interpreter vs compiler (in the context of workflows). Yes Airavata historically took the interpreter approach, where after execution of each node in a workflows, the execution comes back to the enactment engine and re-inspects the state. This facilitated user interactively through executions. Attached state transition diagram might illustrate it more. 

Back to the current scope, I think you got the over all goal correct and your approach is reasonable. There are some details which are missing but thats expected. Just be aware that if your project is accepted, you will need to work with airavata community over the summer and refine the implementation details as you go. You are on a good start. 

Cheers,
Suresh


On Mar 25, 2018, at 8:44 PM, DImuthu Upeksha <dimuthu.upeksha2@gmail.com> wrote:

Hi Yasas,

I'm not a expert in XBaya design and use cases but I think Suresh can shed some light about it. However we no longer use XBaya for workflow interpretation. So don't get confused with the workflows defined in XBaya with the description provided in the JIRA ticket. Let's try to make the concepts clear. We need two levels of Workflows. 

1. To run a single experiment of an Application. We call this as a DAG. So a DAG is statically defined. It can have a set of environment setup tasks, data staging tasks and a job submission task. For example, a DAG is created to run a  Gaussian experiment on a compute host
2. To make a chain of Applications. This is what we call an actual Workflow. In a workflow you can have a Gaussian Experiment running and followed by a Lammps Experiment. So this is a dynamic workflow. Users can come up with different combinations of Applications as a workflow

However your claim is true about pausing and restarting workflows. Either it is a statically defined DAG or a dynamic workflow, we should be able to do those operations.

I can understand some of the words and terminologies that are in those resources are confusing and unclear so please feel free to let us know if you need anything to be clarified.

Thanks
Dimuthu

On Sun, Mar 25, 2018 at 2:45 AM, Yasas Gunarathne <yasasgunarathne@gmail.com> wrote:
Hi All,

I have few questions to be clarified regarding the user-defined workflow execution in Apache Airavata. Here I am talking about the high level workflows that are used to chain together multiple applications. This related to the issue - Airavata-2717 [1].

In this [2] documentation it says that, the workflow interpreter that worked with XBaya provided an interpreted workflow execution framework rather than the compiled workflow execution environments, which allowed the users to pause the execution of the workflow as necessary and update the DAG’s execution states or even the DAG itself and resume execution.

I want to know the actual requirement of having an interpreted workflow execution at this level. Is there any domain level advantage in allowing users to modify the order of workflow at runtime?

I think we can have, pause, resume, restart, and stop commands available even in a compiled workflow execution environment, as long as we don't need to change the workflow.

[1] https://issues.apache.org/jira/browse/AIRAVATA-2717
[2] http://airavata.apache.org/architecture/workflow.html

Regards
--
Yasas Gunarathne
Undergraduate at Department of Computer Science and Engineering
Faculty of Engineering - University of Moratuwa Sri Lanka
LinkedIn | GitHub | Mobile : +94 77 4893616






--
Yasas Gunarathne
Undergraduate at Department of Computer Science and Engineering
Faculty of Engineering - University of Moratuwa Sri Lanka
LinkedIn | GitHub | Mobile : +94 77 4893616