airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Devjyoti Patra <devjyo...@qubole.com>
Subject Re: Q1 Airflow Bay Area Meetup
Date Wed, 31 Jan 2018 08:22:39 GMT
We can present our internal tool for doing data quality checks for ETL's in
Qubole.
Iris is an Airflow Operator that currently has built-in functions for
 count validation,
data-profile validation and duplicate data validation.

On Wed, Jan 31, 2018 at 3:10 AM, George Leslie-Waksman <
george@cloverhealth.com.invalid> wrote:

> If you're still looking for speakers, I have a handful of things I could
> pull together to talk about:
>
>    - the interplay between pools, queues, parallelism, dag concurrency, and
>    other job scheduling levers
>    - layering additional instrumentation on top of Airflow operators
>    - integrity testing DAGs in CI before they hit production / unit testing
>    tasks
>    - using Airflow metadata to track systemic execution behavior
>    - how I would have used Airflow if I'd known then what I know now
>
>
> On Thu, Jan 11, 2018 at 12:18 PM Andrew Maguire <andrewm4894@gmail.com>
> wrote:
>
> > +1 on being able to watch a recording, based in Dublin, Ireland but also
> > very interested in the cloud side of airflow.
> >
> > On Thu, 11 Jan 2018, 17:59 Joy Gao, <joyg@wepay.com> wrote:
> >
> > > Great to see that there's a lot of interests!
> > >
> > > I will go ahead and work on organizing this event. Details TBD.
> > >
> > > On Wed, Jan 10, 2018 at 8:03 AM, Laura Lorenz <
> llorenz@industrydive.com>
> > > wrote:
> > >
> > > > I would love to see videos from this as well, as we're east coast. We
> > > host
> > > > our airflow installation in GCP using GKE, and would love to compare
> > > notes.
> > > >
> > > > Laura
> > > >
> > > > On Tue, Jan 9, 2018 at 4:57 AM, Bolke de Bruin <bdbruin@gmail.com>
> > > wrote:
> > > >
> > > > > If it coincides with qcon.ai (april 10-11) or is close to its
> dates
> > > > then I
> > > > > can join.
> > > > >
> > > > > B.
> > > > >
> > > > > Op 9 jan. 2018 5:36 a.m. schreef "Ananth Durai" <
> vananth22@gmail.com
> > >:
> > > > >
> > > > > I can give a talk about all the hacks we did to scale Airflow Local
> > > > > Executor and improve the data pipeline on-call experience at Slack
> if
> > > > folks
> > > > > are interested.
> > > > >
> > > > > Regards,
> > > > > Ananth.P,
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On 8 January 2018 at 15:51, George Leslie-Waksman <
> > > > > george@cloverhealth.com.invalid> wrote:
> > > > >
> > > > > > +1 and would love to hear about other folks running Airflow
in
> GCP.
> > > > > >
> > > > > > Would strongly prefer March (but I'm just one person)
> > > > > >
> > > > > > On Mon, Jan 8, 2018 at 2:30 PM Feng Lu <fenglu@google.com.invalid
> >
> > > > > wrote:
> > > > > >
> > > > > > > +1 Joy!
> > > > > > >
> > > > > > > Would prefer April if possible, I can talk about running
> Airflow
> > in
> > > > GCP
> > > > > > if
> > > > > > > there's sufficient interest.
> > > > > > >
> > > > > > > On Fri, Jan 5, 2018 at 12:59 PM, Sid Anand <sanand@apache.org>
> > > > wrote:
> > > > > > >
> > > > > > > > Sounds great Joy!
> > > > > > > >
> > > > > > > > I've promoted you to *Event Organizer* on the Bay
Area Apache
> > > > Airflow
> > > > > > > > Meetup as per instructions/guidelines on
> > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Meetups
> > > > > > > >
> > > > > > > > Go ahead and start setting up the meetup... we recommend
3
> > > speakers
> > > > > > with
> > > > > > > > 1-2 of them being external to the hosting company.
> > > > > > > >
> > > > > > > > Once the meetup date is setup, we can add it to
> > > > > > > >
> > > https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements
> > > > &
> > > > > > > tweet
> > > > > > > > it or share it over the dev list.
> > > > > > > > -s
> > > > > > > >
> > > > > > > > On Fri, Jan 5, 2018 at 12:10 PM, Joy Gao <joyg@wepay.com>
> > wrote:
> > > > > > > >
> > > > > > > > > Hi folks,
> > > > > > > > >
> > > > > > > > > I'm Joy from WePay. At the last Airflow meetup
in December
> > > there
> > > > > was
> > > > > > a
> > > > > > > > > demand for hosting a future meetup to cover topics
on
> Airflow
> > > > > > > > integrations
> > > > > > > > > with GCP/AWS (for example, CI/CD with GCP/AWS
> > hooks/operators,
> > > > best
> > > > > > > > > practices on running Airflow in the cloud, managed
Airflow,
> > > etc.)
> > > > > > > > >
> > > > > > > > > If there is enough interest, WePay can host the
next event
> > > > > sometimes
> > > > > > in
> > > > > > > > > March/April. And if you would like to give a
talk or have
> > > another
> > > > > > topic
> > > > > > > > > covered, feel free to suggest them as well.
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Joy
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message