airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] ashb commented on a change in pull request #3989: [AIRFLOW-1945] Autoscale celery workers for airflow added
Date Wed, 03 Oct 2018 08:48:56 GMT
ashb commented on a change in pull request #3989: [AIRFLOW-1945] Autoscale celery workers for
airflow added
URL: https://github.com/apache/incubator-airflow/pull/3989#discussion_r222229674
 
 

 ##########
 File path: airflow/config_templates/default_airflow.cfg
 ##########
 @@ -349,6 +349,14 @@ celery_app_name = airflow.executors.celery_executor
 # your worker box and the nature of your tasks
 worker_concurrency = 16
 
+# The minimum and maximum concurrency that will be used when starting workers with the
+# "airflow worker" command. This defines the number of task instances that
+# a worker will take, minimum numbers are created in beginning and scaled up and down
+# based on number of queued tasks. pick these numbers based on resources on
+# worker box and the nature of the task. If autoscale option is available worker_concurrency
+# will be ignored
+worker_autoscale = 12,16
 
 Review comment:
   We probably shouldn't set a default value for this and worker_concurrency. Please comment
this one out.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message