airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] yrqls21 commented on a change in pull request #3830: [AIRFLOW-2156] Parallelize Celery Executor
Date Wed, 05 Sep 2018 01:09:15 GMT
yrqls21 commented on a change in pull request #3830: [AIRFLOW-2156] Parallelize Celery Executor
URL: https://github.com/apache/incubator-airflow/pull/3830#discussion_r215110793
 
 

 ##########
 File path: airflow/config_templates/default_airflow.cfg
 ##########
 @@ -380,6 +380,9 @@ flower_port = 5555
 # Default queue that tasks get assigned to and that worker listen on.
 default_queue = default
 
+# How many processes CeleryExecutor uses to sync task state.
+sync_parallelism = 16
 
 Review comment:
   I can tune it back to 1 but I can hardly call it backward compatible, since we are using
subprocess to fetch and the functionality doesn't actually change. About the test env, we
were running 16 as mentioned--sry for missing that.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message