airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vincent Poulain <vincent.poul...@tinyclues.com>
Subject Dynamic DAG Generation
Date Sat, 27 Aug 2016 19:05:39 GMT
Hello,
First, thank you for your great job. Airflow made my job easier :)

Is a dynamic generation of DAG a good practice ? As WePay explained there :
https://wecode.wepay.com/posts/airflow-wepay.

In part of a DAG, I have *n* ECS tasks need to be launched daily, each one
with specific *params*. *n & params* depending on configuration which is
daily updated.
For now, I have just one task fetching this configuration and internally
broadcast ECS task.
I am wondering if Airflow would be suitable to generate DAG dynamically
(depending on configuration) in order to split this task into those
multiple ECS task (using ECSOperator, for instance).

Thank you for your tips,

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message