airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael O. (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AIRFLOW-1945) Pass --autoscale to celery workers
Date Wed, 20 Dec 2017 18:06:00 GMT
Michael O. created AIRFLOW-1945:
-----------------------------------

             Summary: Pass --autoscale to celery workers
                 Key: AIRFLOW-1945
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1945
             Project: Apache Airflow
          Issue Type: Improvement
          Components: celery, cli
            Reporter: Michael O.
            Priority: Trivial


Celery supports autoscaling of the worker pool size (number of tasks that can parallelize
within one worker node).  I'd like to propose to support passing the --autoscale parameter
to {{airflow worker}}.
Since this is a trivial change, I am not sure if there's any reason for not being supported
already.(?)

For example
{{airflow worker --concurrency=4}} will set a fixed pool size of 4.

With minimal changes in [https://github.com/apache/incubator-airflow/blob/4ce4faaeae7a76d97defcf9a9d3304ac9d78b9bd/airflow/bin/cli.py#L855]
it could support
{{airflow worker --autoscale=2,10}} to set an autoscaled pool size of 2 to 10


Some references:
* http://docs.celeryproject.org/en/latest/internals/reference/celery.worker.autoscale.html
* https://github.com/apache/incubator-airflow/blob/4ce4faaeae7a76d97defcf9a9d3304ac9d78b9bd/airflow/bin/cli.py#L855



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message