airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dan Davydov (JIRA)" <>
Subject [jira] [Created] (AIRFLOW-2156) Parallelize Celery Executor
Date Tue, 27 Feb 2018 23:48:00 GMT
Dan Davydov created AIRFLOW-2156:

             Summary: Parallelize Celery Executor
                 Key: AIRFLOW-2156
             Project: Apache Airflow
          Issue Type: Improvement
          Components: celery
            Reporter: Dan Davydov
            Assignee: Dan Davydov

The CeleryExecutor doesn't currently support parallel execution to check task states since
Celery does not support this. This can greatly slow down the Scheduler loops since each
request to check a task's state is a network request.


The Celery Executor should parallelize these requests.

This message was sent by Atlassian JIRA

View raw message