airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Viraj Parekh <vi...@astronomer.io>
Subject Re: Bay Area Airflow meetup
Date Fri, 10 May 2019 17:44:45 GMT
Hi Feng,

I'd also like to submit a talk proposal for the Bay Area meetup, works out
nicely that I'll be in town that week.

Title: Running Apache Airflow Reliably on Kubernetes (and other open source
software)

Description: As companies scale up their Airflow usage, they need more
control and observability. By deploying an Airflow stack via Helm on
Kubernetes, fresh environments can be easily spun up or down, and can scale
to near 0 when no jobs are running. This talk will go through the technical
challenges to support thousands of Airflow deployments, how to monitor
them, reliably push DAGs, and build supporting cloud-native infrastructuret
using open-source.

About me: I'm a Senior Data Engineer at Astronomer where I've been working
with Airflow for 2.5 years, and also a Venture for America Fellow. I spend
most of my time with customer implementations (in all clouds), and
providing customized Airflow training

On Fri, May 10, 2019 at 1:15 PM Gabriel Silk <gsilk@dropbox.com.invalid>
wrote:

> Hey Feng,
>
> I'm the tech lead on Data Infrastructure at Dropbox, and I'd like to talk
> about the challenges and opportunities in migrating large-scale ETL to
> Airflow from a system that was built in-house.
>
>
> On Fri, May 10, 2019 at 8:23 AM Szymon Przedwojski <
> szymon.przedwojski@polidea.com> wrote:
>
> > Hi Feng,
> >
> > I'd like to submit my talk proposal for the Bay Area meetup.
> >
> > *Working title:* Migrating Apache Oozie Workflows to Apache Airflow - the
> > developer’s perspective
> >
> > I want to talk about the Oozie-to-Airflow [1] project created by Google
> > that we at Polidea help develop.
> > It's a conversion tool written in Python that generates Airflow Python
> DAGs
> > from Oozie workflow XMLs, thus enabling easy migration of Oozie workflows
> > to Airflow.
> > I would like to present the architecture of the solution and of the whole
> > end-to-end system tests suite, talk about the challenges we faced when
> > translating various Oozie features to Airflow and discuss the future of
> the
> > project.
> > Moreover, I will show how the Airflow community can get involved in the
> > open-source Oozie-to-Airflow project and why it would be mutually
> > beneficial.
> >
> > *About me:*
> > I’m a Software Engineer @ Polidea in Warsaw. I’ve been programming
> > professionally for 5 years now, mainly using Python, Java and JS.
> > I have public speaking experience - I gave a couple of talks at Python
> and
> > JS meetups, at a GDG Dev Fest Warsaw event and multiple internal talks at
> > companies I’ve worked for.
> > I’ve been involved with Airflow since August last year and I’ve
> contributed
> > several operators for different GCP services.
> >
> > [1] https://github.com/GoogleCloudPlatform/cloud-composer
> >
> > *Szymon Przedwojski*
> > Polidea | Software Engineer
> >
> > M: +48 500 330 790
> >
> >
> > On Thu, 18 Apr 2019 at 01:08, Feng Lu <fenglu@google.com.invalid> wrote:
> >
> > > We've finalized the meetup logistics (details here
> > > <
> > >
> >
> https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/260712102/
> > > >).
> > > Please send in your talk proposals!
> > > Thank you.
> > >
> > > Feng
> > >
> > > On Tue, Mar 12, 2019 at 1:15 PM Feng Lu <fenglu@google.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > It has been a while since we had the last Bay Area meetup and we
> > (Google)
> > > > are going to host another meetup around June this year. Will share
> with
> > > > everyone the logistic details, meanwhile, please feel free to send in
> > > your
> > > > talk proposals!
> > > > (I see a lot of very interesting AIP ideas floating around, these are
> > > > great candidates.)
> > > >
> > > > Looking forward to seeing you all at the event ;)
> > > >
> > > > Feng
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message