airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Goretsky <robert.goret...@gmail.com>
Subject Re: Deploy Airflow on Kubernetes using Airflow Operator
Date Mon, 13 Aug 2018 00:17:43 GMT
Barni,
Thank you so much for sharing this!  I'm admittedly far from a Kubernetes
guru, but I'm just trying to wrap my head around the reasons why we'd need
a custom Kubernetes controller to manage Airflow's components, as opposed
to the setup here https://github.com/mumoshu/kube-airflow wherein we simply
deploy Airflow to Kubernetes using standard Kubernetes Services and
Deployments.  I assume you're doing this because, as you mention in the
"Design" section of your repo, "In case of some stateful applications, the
declarative models provided by kubernetes are not sufficient to handle
fault remediation, scaling with data integrity and availability".   Can you
be more specific about what kinds of faults / scaling issues your
Kubernetes Controller might handle for Airflow that otherwise would not be
handled by built-in Kubernetes controllers (Services/Deployments)?

Thanks,
Rob



On Mon, Aug 6, 2018 at 1:55 AM Bolke de Bruin <bdbruin@gmail.com> wrote:

> Really awesome stuff. We are in progress to move over to k8s for Airflow
> (on prem though) and this is really helpful.
>
> B.
>
> Verstuurd vanaf mijn iPad
>
> > Op 3 aug. 2018 om 23:35 heeft Barni Seetharaman <barni@google.com.INVALID>
> het volgende geschreven:
> >
> > Hi
> >
> > We at Google just open-sourced a Kubernetes custom controller (also
> called
> > operator) to make deploying and managing Airflow on kubernetes simple.
> > The operator pattern is a power abstraction in kubernetes.
> > Please watch this repo (in the process of adding docs) for further
> updates.
> >
> > https://github.com/GoogleCloudPlatform/airflow-operator
> >
> > Do reach out if you have any questions.
> >
> > Also created a channel in kubernetes slack  (#airflow-operator)
> > <https://kubernetes.slack.com/messages/CC1UAMYSV/details/> for any
> > discussions specific to Airflow on Kubernetes (including Daniel's
> > Kubernetes Executor, Kuberenetes operator and this custom controller also
> > called kuberntes airflow operator).
> >
> > regs
> > Barni
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message