airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Riccomini <criccom...@apache.org>
Subject Re: Voting Changes for Scheduler-related PRs/Commits
Date Thu, 12 May 2016 22:49:12 GMT
Hey all,

Perhaps we could work backwards from the goal of "I don't want AirBNB's
production Airflow installation broken."

I have observed a few things, personally, that I think we could work
towards fixing:

1) Instability on master
2) Lack of code reviews (until it's too late) by those that care
3) Lack of design docs

For (1), I have a few thoughts. One is better test coverage (duh). Perhaps
we can lay out some rules for when tests are required before a PR will get
merged, etc. A second thought is related to the way we release Airflow. If
you look at how the Linux Kernel [
http://stackoverflow.com/questions/3177338/how-is-linux-kernel-tested], for
example, we could follow something like that.

For (3), I think Jeremiah's work on the dagrun refactor (
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=62694286)
and scheduler 2.0 work is an awesome example. IMO, an large scale work
should have a design doc beforehand. This will prevent things from getting
committed that folks don't agree with.

Cheers,
Chris

On Thu, May 12, 2016 at 3:14 PM, Jakob Homan <jghoman@gmail.com> wrote:

> Dan-
>    I'm afraid not, since that's just a way evading the Apache Way
> rather than working towards it, as Incubation is meant to do.  (Here's
> a good presentation for those unfamiliar with this manta:
> http://www.slideshare.net/gagravarr/the-apache-way-apachecon-2014
> You'll hear it a lot here).
>    The concern here is that bad code may make it into the source tree,
> and eventually into a release.  First, I'd say, yeah, that'll happen.
> All code's reversible, so we can't guarantee it won't happen.  But
> there are a few things the community can do:
> * Maintain a devel branch that interested parties could run off of.
> This would give time for features to be tested before being merged to
> master.  Some policy of merging devel to master could be in place.
> * Switch to a Commit-then-Review model (any committer can commit a
> patch without a +1; this makes reverting bad commits easy and routine
> without the drama/conflict often associated with reverts in
> Review-then-Commit projects).
> * Improve test coverage and utilize ASF and Github resources for testing.
>
> What we can't do is tie abilities/privileges/responsibilities to
> companies (or only people who work for certain companies).  The big
> goal of Incubation is to develop a healthy community around the code
> that can survive and thrive even if one group of contributors
> disappear (say, if a company decides to pull people off the project).
> This is why you'll often see big, big arguments around PMC/podling
> diversity flare up (e.g. http://bit.ly/ASFWayDiversityArgument).
>
> -Jakob
>
> On 12 May 2016 at 14:47, Dan Davydov <dan.davydov@airbnb.com.invalid>
> wrote:
> > @Jakob
> > What if we made it more generic, e.g. a +1 from any commiter from a
> company
> > that is running at a certain scale (e.g. at least X workers) and willing
> to
> > help stage releases in their prods until we have more comprehensive test
> > coverage/an open source staging environment? This is in Airflow's best
> > interests as otherwise stability will suffer.
> >
> > On Thu, May 12, 2016 at 1:44 PM, Chris Riccomini <criccomini@apache.org>
> > wrote:
> >
> >> @Sid, perhaps defining a cool-off window before a scheduler change can
> be
> >> committed. That way, everyone that cares can have a look at it? Also,
> >> having more than one +1 seems OK with me for scheduler changes. We will
> >> have to decide what "scheduler change" means, though.
> >>
> >> On Thu, May 12, 2016 at 1:39 PM, Jakob Homan <jghoman@gmail.com> wrote:
> >>
> >> > Hey Sid-
> >> >    Thanks for the discussion.  It's a good chance to the new
> >> > contributors to get more experience with the ASF.
> >> >
> >> >    Unfortunately, what you propose is not possible in ASF.  As a
> >> > meritocracy, ASF does not recognize individual's employers (or lack
> >> > thereof).  Merit is earned by the individual and follows them as they
> >> > move from organization to organization.  This is true even for
> >> > podlings.  Employees of certain organizations are not given extra
> >> > power over a project or vote due to their relationship with the
> >> > employer.
> >> >
> >> >    ASF does recognize that at times people will be representing their
> >> > employer (with my $EMPLOYER hat on, is a common way of expressing
> >> > this), but expects that everyone is acting in the best interest of the
> >> > project.
> >> >
> >> > -Jakob
> >> >
> >> > On 12 May 2016 at 12:58, Siddharth Anand <sanand@apache.org> wrote:
> >> > > Hi Folks!As many of you know, Apache Airflow (incubating) came from
> >> > Airbnb, where it currently still represents the largest Airflow
> >> deployment.
> >> > Airflow entered the Apache Incubator shortly over a month ago but
> still
> >> > depends on Airbnb's production deployment to vet its release
> candidates.
> >> As
> >> > Airflow's adoption increases, we expect to leverage multiple
> companies in
> >> > conjunction with Apache Infra resources to vet some of the more
> >> performance
> >> > critical pieces of the code base (e.g. scheduler). We're not there
> yet.
> >> > > So, for future commits and PRs involving the scheduler (and possibly
> >> > other components, e.g. executors), I propose a 2 vote system : at
> least 1
> >> > vote from an Airbnb committer and at least 1 vote from a non-Airbnb
> >> > committer, separate from the PR author. This will more readily
> stabilize
> >> > the Airbnb production system that we rely on to vet and cut releases,
> >> > speeding up our release cycle.
> >> > > Please share your thoughts on the matter along with a vote
> for/against.
> >> > > -s
> >> >
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message