airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "George Leslie-Waksman (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AIRFLOW-1491) Celery executor restarts result in duplicate tasks
Date Mon, 07 Aug 2017 18:06:00 GMT
George Leslie-Waksman created AIRFLOW-1491:
----------------------------------------------

             Summary: Celery executor restarts result in duplicate tasks
                 Key: AIRFLOW-1491
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1491
             Project: Apache Airflow
          Issue Type: Bug
            Reporter: George Leslie-Waksman
            Assignee: George Leslie-Waksman


Since the Celery executor uses internal state to determine what it has queued already, anything
that wipes the executor state (such as a scheduler restart) will result in rescheduling of
tasks.

This bug exacerbates a number of task scheduling race conditions and can result in duplicate
task execution.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message