airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cyril Scetbon <>
Subject Handling running tasks
Date Wed, 06 Jul 2016 21:50:15 GMT

I have dags with tasks that use same configuration and same schedule time frequencies.

When I have x tasks in a dag they run in //. My dag is scheduled every 15 minutes, but sometimes
I have at least a task that runs for more than 15 minutes and 2 identical tasks should not
run at the same time (in my case) but it's what happens.

So when the dag is kicked off (every 15 min) I need Airflow to run tasks only for those that
are not running, which means :

- if at time t1 tasks tk1 is running and tk2, ..., tkn are not running, I need Airflow to
run only tk2,.., tkn but not tk1 cause it's already running
- if at time t2 tasks tk1, ..., tkn are not running, I need Airflow to run tk1,.., tkn

I already tried using depends_on_past=True, however I need failed tasks to be kicked off,
cause I can get some temporary issues.

I also use an upstream task connected to all tasks in my dag to be able to run manually all
tasks if I need to. (I don't use a frequency of 15 minutes in all dags)

Any idea ? 
View raw message