airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Austin Weaver <aus...@flyrlabs.com>
Subject Re: Kubernetes executor production readiness?
Date Wed, 26 Jun 2019 16:13:58 GMT
We attempted to migrate our existing production airflow (celery executor)
to k8 executor, and saw many issues scaling tasks ie. Couldn't run 100+
tasks concurrently, as described here
https://issues.apache.org/jira/browse/AIRFLOW-4346 .

Although there has not been much movement on the ticket, I am curious if
people haven't seen these issues running an airflow k8 executor deployment
at scale?

Thanks,
Austin

On Wed, Jun 26, 2019 at 2:35 AM Ry Walker <ry@rywalker.com> wrote:

> If tasks are short, the k8s startup/shutdown time can be a negative factor.
>
> The nice thing about k8s executor is that you can redeploy airflow without
> the need to drain the system to ensure work won’t be affected.
>
> Also, going all k8s gives you autoscaling.
>
> If you’d like to try our both, you can install trial of Astronomer
> platform in your k8s - it lets you easily switch between executors.
>
> Sent via Superhuman iOS ( https://sprh.mn/?vip=ry@rywalker.com )
>
> On Wed, Jun 26 2019 at 7:29 AM, < qph@scribd.com > wrote:
>
> >
> >
> >
> > Hi all,
> >
> >
> >
> > We have made the decision to migrate all our existing ETL jobs to
> Airflow.
> > Right now, we are deciding whether celery or kubernetes executor would be
> > a better fit.
> >
> >
> >
> > Is anyone currently using kubernetes executor in production? If so, how
> > well does it scale and how stable is it compared to celery executor? We
> > are planning to run about 2000 tasks per day to start with.
> >
> >
> >
> > From what I have seen so far, with kubernetes executor, we will be able
> to
> > restrict resource usage for each task run. Can anyone share some insights
> > on what other advantages does celery executor provide over kubernetes
> > executor except maturity?
> >
> >
> >
> > Thanks,
> > QP
> >
> >
> >



-- 
Austin Weaver
Software Engineer
FLYR, Inc.   www.flyrlabs.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message