airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <>
Subject Re: Voting Changes for Scheduler-related PRs/Commits
Date Fri, 13 May 2016 04:56:31 GMT

I think we can approach this a bit differently. Apache is indeed a meritocracy, ie. based
on the ability and talents of the individuals within the community. I don’t think this means
every committer has the same set of abilities and talents. For example, although I consider
myself very talented (…) I don’t think I have the ability to assess all consequences I
make to the scheduler. Here I would like to have Max, Dan, Paul and/or Jeremiah involved.

So here in lies the solution I think. For changes to the core components scheduler and executor
I suggest the following. Large changes: +1 required from 2 persons from the following list:
Dan, Max, Paul, Jeremiah. Minor changes: +1 from Dan, Max, Paul, Jeremiah, Sid, Bolke. The
one who proposes the change is excluded from voting. In order to increase diversity these
lists are reviewed by the community at least every 6 months. 

We would catch two birds with one stone: It addresses the concern of stability in the core
and we have a path by which we can increase diversity (and we set the first step by including

Obviously, by having better tests and improved coverage we can lower the bar (ie. the required
ability) to make changes to the core components. We are not there yet, so I would invite anyone
aspiring to make changes to the core to write a couple of tests first ;-).

What do you think?


Firstly, we are indeed building an Apache community. This involves a company, in this case

> Op 13 mei 2016, om 00:14 heeft Jakob Homan <> het volgende geschreven:
> Dan-
>   I'm afraid not, since that's just a way evading the Apache Way
> rather than working towards it, as Incubation is meant to do.  (Here's
> a good presentation for those unfamiliar with this manta:
> You'll hear it a lot here).
>   The concern here is that bad code may make it into the source tree,
> and eventually into a release.  First, I'd say, yeah, that'll happen.
> All code's reversible, so we can't guarantee it won't happen.  But
> there are a few things the community can do:
> * Maintain a devel branch that interested parties could run off of.
> This would give time for features to be tested before being merged to
> master.  Some policy of merging devel to master could be in place.
> * Switch to a Commit-then-Review model (any committer can commit a
> patch without a +1; this makes reverting bad commits easy and routine
> without the drama/conflict often associated with reverts in
> Review-then-Commit projects).
> * Improve test coverage and utilize ASF and Github resources for testing.
> What we can't do is tie abilities/privileges/responsibilities to
> companies (or only people who work for certain companies).  The big
> goal of Incubation is to develop a healthy community around the code
> that can survive and thrive even if one group of contributors
> disappear (say, if a company decides to pull people off the project).
> This is why you'll often see big, big arguments around PMC/podling
> diversity flare up (e.g.
> -Jakob
> On 12 May 2016 at 14:47, Dan Davydov <> wrote:
>> @Jakob
>> What if we made it more generic, e.g. a +1 from any commiter from a company
>> that is running at a certain scale (e.g. at least X workers) and willing to
>> help stage releases in their prods until we have more comprehensive test
>> coverage/an open source staging environment? This is in Airflow's best
>> interests as otherwise stability will suffer.
>> On Thu, May 12, 2016 at 1:44 PM, Chris Riccomini <>
>> wrote:
>>> @Sid, perhaps defining a cool-off window before a scheduler change can be
>>> committed. That way, everyone that cares can have a look at it? Also,
>>> having more than one +1 seems OK with me for scheduler changes. We will
>>> have to decide what "scheduler change" means, though.
>>> On Thu, May 12, 2016 at 1:39 PM, Jakob Homan <> wrote:
>>>> Hey Sid-
>>>>   Thanks for the discussion.  It's a good chance to the new
>>>> contributors to get more experience with the ASF.
>>>>   Unfortunately, what you propose is not possible in ASF.  As a
>>>> meritocracy, ASF does not recognize individual's employers (or lack
>>>> thereof).  Merit is earned by the individual and follows them as they
>>>> move from organization to organization.  This is true even for
>>>> podlings.  Employees of certain organizations are not given extra
>>>> power over a project or vote due to their relationship with the
>>>> employer.
>>>>   ASF does recognize that at times people will be representing their
>>>> employer (with my $EMPLOYER hat on, is a common way of expressing
>>>> this), but expects that everyone is acting in the best interest of the
>>>> project.
>>>> -Jakob
>>>> On 12 May 2016 at 12:58, Siddharth Anand <> wrote:
>>>>> Hi Folks!As many of you know, Apache Airflow (incubating) came from
>>>> Airbnb, where it currently still represents the largest Airflow
>>> deployment.
>>>> Airflow entered the Apache Incubator shortly over a month ago but still
>>>> depends on Airbnb's production deployment to vet its release candidates.
>>> As
>>>> Airflow's adoption increases, we expect to leverage multiple companies in
>>>> conjunction with Apache Infra resources to vet some of the more
>>> performance
>>>> critical pieces of the code base (e.g. scheduler). We're not there yet.
>>>>> So, for future commits and PRs involving the scheduler (and possibly
>>>> other components, e.g. executors), I propose a 2 vote system : at least 1
>>>> vote from an Airbnb committer and at least 1 vote from a non-Airbnb
>>>> committer, separate from the PR author. This will more readily stabilize
>>>> the Airbnb production system that we rely on to vet and cut releases,
>>>> speeding up our release cycle.
>>>>> Please share your thoughts on the matter along with a vote for/against.
>>>>> -s

View raw message