airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Imberman <>
Subject Re: Current state of airflow on kubernetes
Date Wed, 10 Apr 2019 22:04:11 GMT
Hi all,

Will try to answer questions one-by-one:

*Is the k8s executor stable?*
I've spoken to multiple companies that run the k8s executor in production
and have not heard any stories of major issues. It's a feature that's still
actively in development, but it has a pretty good track record for
stability. uses a lightly modified version of the k8s
executor for their on-prem offering so they could speak to their stories
pretty well (cc: @Greg Neiheisel <> )

*What are major benefits of the k8s executor?*
There are a few major advantages to the k8s executor:

1. No need for capacity planning

Unlike other executors where you need to determine and resize the number of
workers requires, the k8s executor dynamically resized. This also prevents
two major scheduling issues: under-utilization and backlog. Preventing
backlog is really valuable if you want all jobs completed in a timely
manner, and preventing under-utilization will reduce your cloud
bill/compute usage.

2. Better fault tolerance:

The k8s executor takes advantage of the kubernetes event stream to ensure
that state is not held locally within the airflow pod. This means if a
scheduler goes down, you can recreate the state of the scheduler by
re-reading this event log. This means your system can go down for
hours/days and pick up right where it left off with no duplication/skipped

3. Custom airflow images/resources per-task

You can launch docker images on a per-task basis to handle a lot of
lower-level dependencies. Say you have a task requiring certain DPKGs or
system libraries, but don't want to lug those libraries everywhere, now
they only exist where needed (i.e. you could have an image with java)

4. Per-task security

You can use k8s secrets to inject credentials on a per-task basis, so you
don't need to have all of your accounts available on the same pods.

To Magnus' point the benefits of the k8s operator vs. executor are pretty
different (though they are complimentary). The point of the k8s executor
isn't that you can launch pods in k8s (the operator can do that just fine),
but it scales really well (you can have hundreds/potentially thousands of
tasks running in parallel and then have the cluster shrink back to a single
pod), offers valuable stability/security features, and is significantly
easier WRT maintenance (no need to monitor celery/redis/, launches from a
helm script, restarts itself on any failures).

Hope this helps :)

On Wed, Apr 10, 2019 at 12:32 PM Kyle Hamlin <> wrote:

> I would really love to know what are the advantages of the
> KubernetesExecutor are over the CeleryExecutor. Also, what are the types of
> use cases that KubernetesExecutor is well suited for vs CeleryExecutor and
> vise versa?
> On Wed, Apr 10, 2019 at 1:09 PM Magnus Runesson <>
> wrote:
> > Hi!
> >
> > I were in similar position early this year at my work at Tink and went
> > with Kubernetes. We do run all our jobs via the Kubernetes pod operator
> > and rely on K8s to schedule resources etc. A big advantage for us is
> > that we reuse same K8s environment as our services uses for our data
> > processing. Airflow has it own namespace and our infrastructure team
> > takes care about K8s. We have not had any stability issues and basically
> > not have to think that much about maintenance. So in my perspective much
> > less than using Celery.
> >
> > Each task running its own pod means we can ha different software for
> > different tasks and we run them as different service accounts with
> > different permissions regarding access to databases, network etc. We
> > have not found any major drawbacks, rather the opposite with the
> > advantages mentioned above.
> >
> > We needed to add some capabilities to control the security context of
> > our pods since our environment is quite locked down. See
> >
> >
> > and
> >
> > We evaluated the Kube executor but decided not to use it. Since our
> > Airflow instance only have to mange tasks that spawns K8s pods we found
> > the Kube executor did not add any benefits for us. The load on the
> > airflow instance to manage our pods is so small. Some drawbacks not
> > using the Kube executor is that we "cannot" use other operators than the
> > Kubernetes operator. But we do not want that since that would remove the
> > isolation we have between tasks. I should say, the tests we did with the
> > Kube executor showed no stability issues.
> >
> > Hope this gives you some input.
> >
> > /Magnus
> >
> >
> > On 2019-04-10 08:53, Ashwin Sai Shankar wrote:
> > > Hi Airflow devs,
> > > I am to new airflow and trying to figure out some details which will
> help
> > > me choose the right deployment for my company. I want to create a
> > > distributed airflow deployment either using Kube or Celery executor,
> and
> > > had a few questions.
> > >
> > > 1. I see that airflow on kube is a recent feature. How stable is this
> > > feature for production deployments in the most recent oss version?  Are
> > > there any important known problems with airflow on kube?
> > >
> > > 2. I notice that some jiras are still pending in the umbrella jira(
> > > Is it better to
> > wait
> > > for these tickets to be closed?
> > >
> > > 3. How does kube deployment compare to Celery in terms of stability,
> > > performance, feature set, ease of deployment, maintenance?
> > >
> > > Thanks,
> > > Ash
> > >
> >
> --
> Kyle Hamlin

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message