airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ry Walker ...@rywalker.com>
Subject Re: Kubernetes executor production readiness?
Date Wed, 26 Jun 2019 16:20:07 GMT
Once Daniel Imberman is back from his vacation, I do know he’s planning to
jump into that JIRA issue to see what is going on.
-Ry


On Wednesday, June 26, 2019, Austin Weaver <austin@flyrlabs.com> wrote:

> We attempted to migrate our existing production airflow (celery executor)
> to k8 executor, and saw many issues scaling tasks ie. Couldn't run 100+
> tasks concurrently, as described here
> https://issues.apache.org/jira/browse/AIRFLOW-4346 .
>
> Although there has not been much movement on the ticket, I am curious if
> people haven't seen these issues running an airflow k8 executor deployment
> at scale?
>
> Thanks,
> Austin
>
> On Wed, Jun 26, 2019 at 2:35 AM Ry Walker <ry@rywalker.com> wrote:
>
> > If tasks are short, the k8s startup/shutdown time can be a negative
> factor.
> >
> > The nice thing about k8s executor is that you can redeploy airflow
> without
> > the need to drain the system to ensure work won’t be affected.
> >
> > Also, going all k8s gives you autoscaling.
> >
> > If you’d like to try our both, you can install trial of Astronomer
> > platform in your k8s - it lets you easily switch between executors.
> >
> > Sent via Superhuman iOS ( https://sprh.mn/?vip=ry@rywalker.com )
> >
> > On Wed, Jun 26 2019 at 7:29 AM, < qph@scribd.com > wrote:
> >
> > >
> > >
> > >
> > > Hi all,
> > >
> > >
> > >
> > > We have made the decision to migrate all our existing ETL jobs to
> > Airflow.
> > > Right now, we are deciding whether celery or kubernetes executor would
> be
> > > a better fit.
> > >
> > >
> > >
> > > Is anyone currently using kubernetes executor in production? If so, how
> > > well does it scale and how stable is it compared to celery executor? We
> > > are planning to run about 2000 tasks per day to start with.
> > >
> > >
> > >
> > > From what I have seen so far, with kubernetes executor, we will be able
> > to
> > > restrict resource usage for each task run. Can anyone share some
> insights
> > > on what other advantages does celery executor provide over kubernetes
> > > executor except maturity?
> > >
> > >
> > >
> > > Thanks,
> > > QP
> > >
> > >
> > >
>
>
>
> --
> Austin Weaver
> Software Engineer
> FLYR, Inc.   www.flyrlabs.com
>


-- 
Sent from Gmail Mobile

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message