airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cyril Scetbon (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AIRFLOW-215) Airflow worker (CeleryExecutor) needs to be restarted to pick up tasks
Date Mon, 06 Jun 2016 19:44:21 GMT
Cyril Scetbon created AIRFLOW-215:
-------------------------------------

             Summary: Airflow worker (CeleryExecutor) needs to be restarted to pick up tasks
                 Key: AIRFLOW-215
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-215
             Project: Apache Airflow
          Issue Type: Bug
    Affects Versions: Airflow 1.7.1.2
            Reporter: Cyril Scetbon


We have a main dag that dynamically creates subdags containing tasks using BashOperator. Using
CeleryExecutor we see Celery tasks been created with *STARTED* status but they are not picked
up by our worker. However, if we restart our worker, then tasks are picked up. 

Here you can find code if you want to try to reproduce it https://www.dropbox.com/s/8u7xf8jt55v8zio/dags.zip.

We also tested using LocalExecutor and everything worked fine.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message