airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [airflow] ddelange commented on issue #3989: [AIRFLOW-1945] Autoscale celery workers for airflow added
Date Fri, 27 Sep 2019 10:20:26 GMT
ddelange commented on issue #3989: [AIRFLOW-1945] Autoscale celery workers for airflow added
URL: https://github.com/apache/airflow/pull/3989#issuecomment-535882666
 
 
   I actually really like the potential of using celery autoscale (e.g. put one massive worker
on an autoscaling AWS EC2 instance to solve issue of scaling airflow temporarily for a sudden
heavy load).
   I saw they added autoscaling back for celery milestones 4.5/4.6/4.7 or else for sure v5,
so I've left the autoscaling option in our airflow config file, linking to the celery issue
in our comments below the comments from `default_config.cfg`, and hope they fix it in future
celery version, while assuming that at some point airflow will bump celery along in a future
version and it'll start working again :)
   
   right now it's not breaking anything having this option, celery only just ignores this
option internally, and permanently puts it on the minimum concurrency specified. so from my
side all good for now. just wanted to let you know (and anyone else who may stumble upon this
PR via git blame) ^^

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message