airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Qingping Hou <...@scribd.com>
Subject Re: Kubernetes executor production readiness?
Date Fri, 28 Jun 2019 23:32:12 GMT
On Wed, Jun 26, 2019 at 2:35 AM Ry Walker <ry@rywalker.com> wrote:
> The nice thing about k8s executor is that you can redeploy airflow without the need to
drain the system to ensure work won’t be affected.
>
> Also, going all k8s gives you autoscaling.
>
> If you’d like to try our both, you can install trial of Astronomer platform in your
k8s - it lets you easily switch between executors.

Being able to redeploy without draining seems like a big advantage to
me. I will check out astronomer platform as well.

On Wed, Jun 26, 2019 at 9:14 AM Austin Weaver <austin@flyrlabs.com> wrote:
>
> We attempted to migrate our existing production airflow (celery executor)
> to k8 executor, and saw many issues scaling tasks ie. Couldn't run 100+
> tasks concurrently, as described here
> https://issues.apache.org/jira/browse/AIRFLOW-4346 .
>

Thanks for the info. Looking at the symptom, it does look like you are
hitting the k8s maximum pod per node restriction. I will keep an eye
on the ticket and try to see if I can reproduce it from my side.

Mime
View raw message