airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael O. (JIRA)" <>
Subject [jira] [Created] (AIRFLOW-1945) Pass --autoscale to celery workers
Date Wed, 20 Dec 2017 18:06:00 GMT
Michael O. created AIRFLOW-1945:

             Summary: Pass --autoscale to celery workers
                 Key: AIRFLOW-1945
             Project: Apache Airflow
          Issue Type: Improvement
          Components: celery, cli
            Reporter: Michael O.
            Priority: Trivial

Celery supports autoscaling of the worker pool size (number of tasks that can parallelize
within one worker node).  I'd like to propose to support passing the --autoscale parameter
to {{airflow worker}}.
Since this is a trivial change, I am not sure if there's any reason for not being supported

For example
{{airflow worker --concurrency=4}} will set a fixed pool size of 4.

With minimal changes in []
it could support
{{airflow worker --autoscale=2,10}} to set an autoscaled pool size of 2 to 10

Some references:

This message was sent by Atlassian JIRA

View raw message