airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scato Eggen (Jira)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-6252) DAG.clear() does not clear certain task instances
Date Fri, 13 Dec 2019 09:50:00 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-6252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Scato Eggen updated AIRFLOW-6252:
---------------------------------
    Description: 
Use case:

When running a task from a unit test, I need to clear the DAG as demonstrated in [https://blog.godatadriven.com/testing-and-debugging-apache-airflow].

Observed behavior:

At first, everything worked as expected, but at some point the task wouldn't start anymore.
Upon inspection, it turned out that:
 * the task instance had status RUNNING but no job_id
 * airflow.models.taskinstance.clear_task_instances skips tasks without a job_id, so it was
keeping the task instance in state RUNNING
 * as a result TaskInstance._check_and_change_state_before_execution started returning False

A factor that could have contributed was that changed the task from having retries=1 to retries=0.
Also, I was stepping through the code with the debugger.

Expected behavior:

DAG.clear() clears all task instances, even the ones that have weird states.

Workaround:

Delete ~/airflow/airflow.db and rerun airflow initdb.

 

  was:
Use case:

When running a task from a unit test, I need to clear the DAG as demonstrated in [https://blog.godatadriven.com/testing-and-debugging-apache-airflow].

Observed behavior:

At first, everything worked as expected, but at some point the task wouldn't start anymore.
Upon inspection, it turned out that the task instance had status RUNNING but no job_id. airflow.models.taskinstance.clear_task_instances
skips tasks without a job_id, so it was keeping the task instance in state RUNNING and as
a result TaskInstance._check_and_change_state_before_execution started returning False.

A factor that could have contributed was that changed the task from having retries=1 to retries=0.
Also, I was stepping through the code with the debugger.

Expected behavior:

DAG.clear() clears all task instances, even the ones that have weird states.

Workaround:

Delete ~/airflow/airflow.db and rerun airflow initdb.

 


> DAG.clear() does not clear certain task instances
> -------------------------------------------------
>
>                 Key: AIRFLOW-6252
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6252
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: DAG
>    Affects Versions: 1.10.3
>            Reporter: Scato Eggen
>            Priority: Major
>
> Use case:
> When running a task from a unit test, I need to clear the DAG as demonstrated in [https://blog.godatadriven.com/testing-and-debugging-apache-airflow].
> Observed behavior:
> At first, everything worked as expected, but at some point the task wouldn't start anymore.
Upon inspection, it turned out that:
>  * the task instance had status RUNNING but no job_id
>  * airflow.models.taskinstance.clear_task_instances skips tasks without a job_id, so
it was keeping the task instance in state RUNNING
>  * as a result TaskInstance._check_and_change_state_before_execution started returning
False
> A factor that could have contributed was that changed the task from having retries=1
to retries=0. Also, I was stepping through the code with the debugger.
> Expected behavior:
> DAG.clear() clears all task instances, even the ones that have weird states.
> Workaround:
> Delete ~/airflow/airflow.db and rerun airflow initdb.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message