airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dan Davydov (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AIRFLOW-316) Always check DB state for Backfill Job execution
Date Wed, 06 Jul 2016 21:35:11 GMT
Dan Davydov created AIRFLOW-316:
-----------------------------------

             Summary: Always check DB state for Backfill Job execution
                 Key: AIRFLOW-316
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-316
             Project: Apache Airflow
          Issue Type: Bug
            Reporter: Dan Davydov
            Assignee: Dan Davydov


Always check DB state and not just the local state for backfill jobs for determining which
task instances have not yet completed execution. This is to avoid potential race conditions
with e.g. two backfill jobs running the same task instance.

At Airbnb we have noticed a race condition with the subdag operator (which uses the backfill
job) where all of the tasks in the subdag complete successfully, but the parent DAG is still
waiting on a task in the subdag to succeed. Resolving this JIRA should fix the subdag operator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message