airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cyril Scetbon (JIRA)" <>
Subject [jira] [Commented] (AIRFLOW-215) Airflow worker (CeleryExecutor) needs to be restarted to pick up tasks
Date Wed, 08 Jun 2016 14:19:20 GMT


Cyril Scetbon commented on AIRFLOW-215:

We tried to add a new worker. It then worked and after a few tries we got the same behavior.
Even restarting part or all workers didn't work. In logs we see : 

 {} INFO - [backfill progress] | waiting: 1 | succeeded: 0 | kicked_off: 1 | failed:
0 | skipped: 0 | deadlocked:

Is there a debug mode [~bolke] to see what's happening ? Do you have any suspicions ?

> Airflow worker (CeleryExecutor) needs to be restarted to pick up tasks
> ----------------------------------------------------------------------
>                 Key: AIRFLOW-215
>                 URL:
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: celery
>    Affects Versions: Airflow
>            Reporter: Cyril Scetbon
> We have a main dag that dynamically creates subdags containing tasks using BashOperator.
Using CeleryExecutor we see Celery tasks been created with *STARTED* status but they are not
picked up by our worker. However, if we restart our worker, then tasks are picked up. 
> Here you can find code if you want to try to reproduce it
> We also tested using LocalExecutor and everything worked fine.

This message was sent by Atlassian JIRA

View raw message