airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Qingping Hou <...@scribd.com>
Subject Kubernetes executor production readiness?
Date Wed, 26 Jun 2019 06:29:55 GMT
Hi all,

We have made the decision to migrate all our existing ETL jobs to
Airflow. Right now, we are deciding whether celery or kubernetes
executor would be a better fit.

Is anyone currently using kubernetes executor in production? If so,
how well does it scale and how stable is it compared to celery
executor? We are planning to run about 2000 tasks per day to start
with.

>From what I have seen so far, with kubernetes executor, we will be
able to restrict resource usage for each task run. Can anyone share
some insights on what other advantages does celery executor provide
over kubernetes executor except maturity?

Thanks,
QP

Mime
View raw message