airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Grant Nicholas (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-1884) Ensure scheduler is crash safe for externally triggered dagruns
Date Mon, 04 Dec 2017 20:27:00 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-1884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Grant Nicholas updated AIRFLOW-1884:
------------------------------------
    Description: 
Orphaned task instances are only reset for dagruns that are both not externally triggered
and not backfilled. This violates the crash safety property of the scheduler, ie) if the scheduler
crashes in the middle of one of these dagruns then tasks can be stuck in the "Queued" state
forever and never executed. 

I found the changeset this regression happened in, it is this one:
https://issues.apache.org/jira/browse/AIRFLOW-1059

This change reverts the special casing logic so that externally triggered dagruns have orphaned
tasks reset on startup of the scheduler. 

  was:
Orphaned task instances are only reset for dagruns that are both not externally triggered
and not backfilled. This violates the crash safety property of the scheduler, ie) if the scheduler
crashes in the middle of one of these dagruns then tasks can be stuck in the "Queued" state
forever and never executed. 

I found the changeset this regression happened in, it is this one:
https://issues.apache.org/jira/browse/AIRFLOW-1059

This change reverts the special casing logic so that all dagruns have orphaned tasks reset
on startup of the scheduler. 


> Ensure scheduler is crash safe for externally triggered dagruns
> ---------------------------------------------------------------
>
>                 Key: AIRFLOW-1884
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1884
>             Project: Apache Airflow
>          Issue Type: Bug
>            Reporter: Grant Nicholas
>            Assignee: Grant Nicholas
>
> Orphaned task instances are only reset for dagruns that are both not externally triggered
and not backfilled. This violates the crash safety property of the scheduler, ie) if the scheduler
crashes in the middle of one of these dagruns then tasks can be stuck in the "Queued" state
forever and never executed. 
> I found the changeset this regression happened in, it is this one:
> https://issues.apache.org/jira/browse/AIRFLOW-1059
> This change reverts the special casing logic so that externally triggered dagruns have
orphaned tasks reset on startup of the scheduler. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message