airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amit Kulkarni <am...@wepay.com>
Subject Re: Airflow + Kubernetes discussion
Date Sat, 22 Jul 2017 21:23:20 GMT
I would like to join as well.


Amit Kulkarni
Site Reliability Engineer
Mobile:  (716)-352-3270

Payments partner to the platform economy

On Sat, Jul 22, 2017 at 9:07 AM, Victor Monteiro <victor.monteiro@ubee.in>
wrote:

> Hi everyone!
>
> I'm new here and I've read some of the discussions about K8s + Airflow. And
> I would be very interested to join this meeting.
>
> But I'm not quiet sure what is this integration. Is it to make it easy to
> deploy Airflow to K8S and have scheduler, webserver and other airflow
> components inside K8S pods or is it to be able to run dags/tasks in
> isolated pod environments to allow temporary isolated environments that
> will be destroyed as soon as the task instance / dag run is over?
>
> At the company I work on here in Brazil, we have deployed Airflow with K8S
> and we are thinking in a way to run dags/tasks in temporary isolated pods
> to allow, mainly, any powerful operator like PythonOperator and
> BashOperator. Something using this
> https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-
> completion/#what-is-a-job
> .
>
> Ps: Sorry if this wasn't the right thread to discuss about ir. If that is
> true, message me and I'll take this message to the right place.
> On Fri, 21 Jul 2017 at 01:55 Maxime Beauchemin <maximebeauchemin@gmail.com
> >
> wrote:
>
> > +1!
> >
> > On Thu, Jul 20, 2017 at 5:53 PM, Feng Lu <fenglu@google.com.invalid>
> > wrote:
> >
> > > Would like to join, please kindly invite me.
> > > Thanks!
> > >
> > > On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <
> > > daniel.imberman@gmail.com>
> > > wrote:
> > >
> > > > Hello everyone,
> > > >
> > > > Recently there's been a fair amount of discussion regarding the
> > > integration
> > > > of airflow with kubernetes. If there is interest I would love to host
> > an
> > > > e-meeting to discuss this integration. I can go over the architecture
> > as
> > > it
> > > > stands right now and would love feedback on
> > > improvements/features/design. I
> > > > could also attempt to get one or two members of google's kubernetes
> > team
> > > to
> > > > join to discuss best practices.
> > > >
> > > > I'm currently thinking that next Thursday at 11AM PST over zoom.us,
> > > though
> > > > if there's strong opinions otherwise I'd be glad to propose other
> > times.
> > > >
> > > > Cheers!
> > > >
> > > > Daniel
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message