airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gerard Toonstra <gtoons...@gmail.com>
Subject Airflow talk @ Coolblue
Date Sun, 29 Oct 2017 10:29:38 GMT
Hi all,

Thursday the 26/10 my employer Coolblue organized a "Behind the Scenes"
event. It is an opportunity for engineers to talk about stuff they work on
and usually they provide two presentations.

This event was about BigData and Processing. As (now) team lead of Data
Platform, I decided to talk about Apache Airflow, which we are now in the
process of migrating to (from Azkaban).

Here are the slides:

https://www.linkedin.com/feed/update/urn:li:activity:6330346647347875840

It is a technical presentation, aimed at informing people who are new to
Airflow what the underlying architecture is and also presenting the why
you'd want to use it in the first place. I based the architectural diagrams
on AWS on the PoC we did some time.

Important takeaway:

Airflow is built around some great design principles, which are the result
of important insights into data processing. These principles result in a
tool, when used correctly according to these principles, to reduce the ETL
effort and maintenance and make time to work on higher level intelligent
work like Machine Learning, Deep Learning and analysis of your data.

It is very similar to the talk I gave at BigData Week London 2017:

https://youtu.be/Ch2AQhOhefw

Rgds,

Gerard

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message