airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxime Beauchemin <maximebeauche...@gmail.com>
Subject Airflow 2.0
Date Thu, 17 Nov 2016 23:19:34 GMT
*This is a brainstorm email thread about Airflow 2.0!*

I wanted to share some ideas around what I would like to do in Airflow 2.0
and would love to hear what others are thinking. I'll compile the ideas
that are shared in this thread in a Wiki once the conversation fades.

-------------------------------------------

First idea, to get the conversation started:

*Breaking down the package*
`pip install airflow-common airflow-scheduler airflow-webserver
airflow-operators-googlecloud ...`

It seems to me like we're getting to a point where having different
repositories and different packages would make things much easier in all
sorts of ways. For instance the web server is a lot less sensitive than the
scheduler, and changes to operators should/could be deployed at will,
independently from the main package. People in their environment could
upgrade only certain packages when needed. Travis builds would be more
targeted, and take less time, ...

Also, the whole current "extra_requires" approach to optional dependencies
(in setup.py) is kind getting out-of-hand.

Of course `pip install airflow` would bring in a collection of sub-packages
similar in functionality to what it does now, perhaps without so many
operators you probably don't need in your environment.

The release process is the main pain-point and the biggest risk for the
project, and I feel like this a solid solution to address it.

Max

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message