airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Flow-based Airflow?
Date Mon, 23 Jan 2017 16:05:45 GMT
Hi All,

I came by a write up of some of the downsides in current workflow management systems like
Airflow and Luigi (http://bionics.it/posts/workflows-dataflow-not-task-deps) where they argue
dependencies should be between inputs and outputs of tasks rather than between tasks (inlets/outlets).

They extended Luigi (https://github.com/pharmbio/sciluigi) to do this and even published a
scientific paper on it: http://jcheminf.springeropen.com/articles/10.1186/s13321-016-0179-6
.

I kind of like the idea, has anyone played with it, any thoughts? I might want to try it in
Airflow.

Bolke
Mime
View raw message