airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ash Berlin-Taylor (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-2761) Parallelize Celery Executor enqueuing
Date Fri, 05 Oct 2018 09:28:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16639538#comment-16639538
] 

Ash Berlin-Taylor commented on AIRFLOW-2761:
--------------------------------------------

[~xnuinside] if you have any spare time taking a detailed look at pr 3873, and preferably
running it somewhere if you could would be amazingly helpful. 

> Parallelize Celery Executor enqueuing
> -------------------------------------
>
>                 Key: AIRFLOW-2761
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2761
>             Project: Apache Airflow
>          Issue Type: Improvement
>            Reporter: Kevin Yang
>            Priority: Major
>
> Currently celery executor enqueues in an async fashion but still doing that in a single
process loop. This can slows down scheduler loop and creates scheduling delay if we have large
# of task to schedule in a short time, e.g. UTC midnight we need to schedule large # of sensors
in a short period.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message