airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ash Berlin-Taylor (JIRA)" <>
Subject [jira] [Commented] (AIRFLOW-2761) Parallelize Celery Executor enqueuing
Date Fri, 05 Oct 2018 09:28:00 GMT


Ash Berlin-Taylor commented on AIRFLOW-2761:

[~xnuinside] if you have any spare time taking a detailed look at pr 3873, and preferably
running it somewhere if you could would be amazingly helpful. 

> Parallelize Celery Executor enqueuing
> -------------------------------------
>                 Key: AIRFLOW-2761
>                 URL:
>             Project: Apache Airflow
>          Issue Type: Improvement
>            Reporter: Kevin Yang
>            Priority: Major
> Currently celery executor enqueues in an async fashion but still doing that in a single
process loop. This can slows down scheduler loop and creates scheduling delay if we have large
# of task to schedule in a short time, e.g. UTC midnight we need to schedule large # of sensors
in a short period.

This message was sent by Atlassian JIRA

View raw message