airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saurabh Agrawal <agras...@umail.iu.edu>
Subject Discarding use of Airflow in the Airavata Workflow Engine
Date Wed, 11 Oct 2017 18:39:46 GMT
Hi all,

I was working on standardizing workflow execution.
The details of this issue could be found in Jira
<https://issues.apache.org/jira/browse/AIRAVATA-2555>.
We are thinking about using CWL
<https://github.com/common-workflow-language/common-workflow-language> to
pass the task DAG information.

We were initially considering using Apache Airflow
<https://airflow.incubator.apache.org/> to trigger the constituent tasks in
the workflow. But, after a brief research, we found the following reasons
to not use it:

   - Airflow cannot work on a time scale lesser than a minute. Thus, we
   cannot execute a workflow immediately.
   - Airflow suggests that tasks should be big enough to run for at least 5
   minutes. The tasks in Airavata workflow might be smaller than that
   - Airflow is mostly used to configure static workflows, whereas we might
   want to create workflows on the fly
   - [Unsure] Airflow requires client installation to be done on the
   executor machines
   - One of Airflow's biggest design goals is to visualize the workflow.
   Since we are executing a task inside Airavata, we might not be able to use
   this feature

Currently, we are exploring other tools for the same task.
Please correct the above rationale if it is incorrect.
Also, please suggest other tools that could solve this problem.

Best Wishes,
Saurabh Agrawal
LinkedIn <https://www.linkedin.com/in/saurabh-agrawal-1a048b118i> Github
<https://github.com/saurabhagrawal0412>

Mime
View raw message