airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vardanguptacse@gmail.com <vardangupta...@gmail.com>
Subject Pause DAG leading to Hang State of DAG
Date Wed, 25 Jul 2018 16:02:41 GMT
Hi Everyone,

We are using Airflow v1.9 and observing a pattern where we have few DAGs marked in paused
state and scheduled existing DAG runs are not getting marked success/failure instead they
are always in running state. 

The moment, we unpause the DAG, it will back fill all the missed executions along with the
changing state of existing DAG runs in dag_run table. Since, we've written a DELETE functionality
in an abstracted form which works only if the DAG is disabled & no active runs are there,
we could not perform DELETE operation as runs are in running state forever in such scenarios.

Is it a default behavior that if a DAG is disabled then no further dag run will be kicked
off along with no execution of residual part of running instances of DAG(may be 2/3 task instances)
which was unprocessed by the time paused state was requested? If this is a bug, we can open
JIRA and also contribute.

Let us know, if additional information is required. Would love to get this addressed.

Mime
View raw message