airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From siddharth anand <>
Subject Priorities for the Airflow Community
Date Fri, 18 Nov 2016 01:41:04 GMT
As we near the last month of the year and the 9th month in Apache
Incubation, I'd like to share some thoughts and gauge community feedback.

At the beginning of this year, we saw a good deal of interest in Airbnb's
Airflow project. As more companies started using it for business critical
work, it became evident that the only way for the project to scale and
improve would be to transfer ownership via Apache Incubation to the user

We started incubation by swiftly adopting Apache processes and tooling,
selecting the initial committer group, and getting familiar with the inner
workings of Airflow.

The first few months saw various committers and contributors dive into the
guts of Airflow's scheduler (Bolke, Jeremiah, and Paul to name just a few).
The result was a lot of bug fixes, code refactoring, & solid performance
gains, followed by a period of stability in the core Airflow code (the
scheduler). We also saw a large influx of PRs from a growing contributor
base that buried the maintainers until recently. From cool new UI and CLI
features to more 3rd party integrations (e.g. GCP, AWS, Docker) to
improvements in logging and documentation, the community has helped make
Airflow more useful to all. Along the way, we have been promoting
contributors to the committer group and PPMC in order to better serve you.
We plan to continue rewarding member of our community with such promotions.

Roughly 8 months after entering incubation, we have seen Airflow grow :

   - from 100 contributors  to 212
   - from 30 companies to ~70
   - by thousands of commits
   - with increased coverage at conferences and literature

As a community, our job is not yet done:

   - We want your continued contribution
      - With so many companies and varied use-cases, your contributions
      what makes Airflow a rich ecosystem
      - We welcome pull requests that improve user experience, expand 3rd
      party integrations, address some quirks and annoyances in Airflow (e.g.
      changing the dag_id in order to recognize start_date changes, running on
      UTC everywhere), or simply improve documentation
   - To support your active involvement as contributors
      - We, the maintainers, are committed to supporting a timely (<2 week)
      turnaround of your Pull Requests
   - To support your active involvement as users
      - We, the maintainers, need to be better at releasing all of these
      contributions at a predictable cadence
         - This works in unison with reviewing PRs in a timely manner
      - We will be focusing on a predictable and frequent release cadence
      the first half of next year.

As more companies join the community, enterprise-level requirements start
to become more important. Features around security, stability, performance,
backward compatibility, and ease of installation / upgrade become more
important. It's a sign of maturity and of our entering the mainstream. We
will continue to accept features but will also consider risk when reviewing
PRs (e.g. potential for destabilization).

If there are other things we need to focus on or to prioritize, please do
share your thoughts.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message