airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dan Davydov (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (AIRFLOW-316) Always check DB state for Backfill Job execution
Date Wed, 06 Jul 2016 21:45:11 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Dan Davydov resolved AIRFLOW-316.
---------------------------------
    Resolution: Fixed

> Always check DB state for Backfill Job execution
> ------------------------------------------------
>
>                 Key: AIRFLOW-316
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-316
>             Project: Apache Airflow
>          Issue Type: Bug
>            Reporter: Dan Davydov
>            Assignee: Dan Davydov
>              Labels: backfill
>
> Always check DB state and not just the local state for backfill jobs for determining
which task instances have not yet completed execution. This is to avoid potential race conditions
with e.g. two backfill jobs running the same task instance.
> At Airbnb we have noticed a race condition with the subdag operator (which uses the backfill
job) where all of the tasks in the subdag complete successfully, but the parent DAG is still
waiting on a task in the subdag to succeed. Resolving this JIRA should fix the subdag operator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message