airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Victor Monteiro <victor.monte...@ubee.in>
Subject Re: Airflow + Kubernetes discussion
Date Sat, 22 Jul 2017 16:07:53 GMT
Hi everyone!

I'm new here and I've read some of the discussions about K8s + Airflow. And
I would be very interested to join this meeting.

But I'm not quiet sure what is this integration. Is it to make it easy to
deploy Airflow to K8S and have scheduler, webserver and other airflow
components inside K8S pods or is it to be able to run dags/tasks in
isolated pod environments to allow temporary isolated environments that
will be destroyed as soon as the task instance / dag run is over?

At the company I work on here in Brazil, we have deployed Airflow with K8S
and we are thinking in a way to run dags/tasks in temporary isolated pods
to allow, mainly, any powerful operator like PythonOperator and
BashOperator. Something using this
https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#what-is-a-job
.

Ps: Sorry if this wasn't the right thread to discuss about ir. If that is
true, message me and I'll take this message to the right place.
On Fri, 21 Jul 2017 at 01:55 Maxime Beauchemin <maximebeauchemin@gmail.com>
wrote:

> +1!
>
> On Thu, Jul 20, 2017 at 5:53 PM, Feng Lu <fenglu@google.com.invalid>
> wrote:
>
> > Would like to join, please kindly invite me.
> > Thanks!
> >
> > On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <
> > daniel.imberman@gmail.com>
> > wrote:
> >
> > > Hello everyone,
> > >
> > > Recently there's been a fair amount of discussion regarding the
> > integration
> > > of airflow with kubernetes. If there is interest I would love to host
> an
> > > e-meeting to discuss this integration. I can go over the architecture
> as
> > it
> > > stands right now and would love feedback on
> > improvements/features/design. I
> > > could also attempt to get one or two members of google's kubernetes
> team
> > to
> > > join to discuss best practices.
> > >
> > > I'm currently thinking that next Thursday at 11AM PST over zoom.us,
> > though
> > > if there's strong opinions otherwise I'd be glad to propose other
> times.
> > >
> > > Cheers!
> > >
> > > Daniel
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message