airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <>
Subject [jira] [Commented] (AIRFLOW-1059) Reset_state_for_orphaned_task should operate in batch for the scheduler
Date Fri, 08 Dec 2017 09:22:01 GMT


ASF subversion and git services commented on AIRFLOW-1059:

Commit 8626186ca8c244386a8a97fcaf6d4221270863da in incubator-airflow's branch refs/heads/master
[;h=8626186 ]

[AIRFLOW-1884][AIRFLOW-1059] Reset orphaned task state for external dagruns

On scheduler startup, orphaned task instances have
their state cleared and are rescheduled to avoid
having tasks that are stuck in a QUEUED state
forever. Previously, this check ignored backfilled
and externally triggered dagruns, meaning that
backfilled and externally triggered dagruns could
have orphaned tasks that are stuck forever. This
changeset removes the special case logic for
externally triggered dagruns, ensuring that
externally triggered dagruns are crash safe. This
same fix cannot be applied to backfilled dagruns,
so for now backfilled dagruns are not crash safe.

Closes #2843 from grantnicholas/AIRFLOW-1884

> Reset_state_for_orphaned_task should operate in batch for the scheduler
> -----------------------------------------------------------------------
>                 Key: AIRFLOW-1059
>                 URL:
>             Project: Apache Airflow
>          Issue Type: Improvement
>            Reporter: Alex Guziel
>            Assignee: Alex Guziel
>             Fix For: 1.9.0
> Scheduler startup is very slow due to resetting state making a query for each dag run.
We should be able to do this in a constant number of queries which will increase scheduler
startup time significantly.

This message was sent by Atlassian JIRA

View raw message