airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From George Leslie-Waksman <waks...@gmail.com>
Subject Re: Guidelines on Contrib vs Non-contrib
Date Mon, 17 Sep 2018 23:50:39 GMT
Given we have a plugin system, could we alternatively move away from
keeping non-core supported code outside of the core project/repo?

It would hugely decrease the surface area of the main repository and
testing infrastructure to get most of the contrib code out to its own place.

Further, it would decrease the committer burden of having to approve/merge
code that is not supposed to be their responsibility.

On Mon, Sep 17, 2018 at 4:37 PM Tim Swast <swast@google.com.invalid> wrote:

> > Individual operators and hooks living in separate repositories on github
> (or possibly other Apache projects), which are then distributed by pip and
> installed as libraries seems like it would scale better.
>
> Pandas did this about a year ago, and it's seemed to have worked well. For
> example, pandas.read_gbq is a very thin wrapper around pandas_gbq.read_gbq
> (distributed as a separate package). It has made it easier for me to track
> issues corresponding to my area of expertise.
>
> On Sun, Sep 16, 2018 at 1:25 PM Jakob Homan <jghoman@gmail.com> wrote:
>
> > > My understanding as a contributor is that if a hook/operator is in
> core,
> > it
> > > means that a committer is willing to take personal responsibility to
> > > maintain it (or at least help maintain it), and everything else goes in
> > > contrib.
> >
> > That's not correct.  All of the code is owned by the entire
> > community[1]; no one person is responsible for any of it.  There's no
> > silos, fiefdoms, walled gardens, etc.  If the community cannot support
> > a piece of code it should be deprecated and subsequently removed.
> >
> > Contrib sections are almost always problematic for this reason.
> > Hadoop ended up abandoning its.  Because Airflow acts as a gathering
> > point for so many disparate technologies (databases, storage systems,
> > compute engines, etc.), trying to keep all of them corralled and up to
> > date will be very difficult.  Individual operators and hooks living in
> > separate repositories on github (or possibly other Apache projects),
> > which are then distributed by pip and installed as libraries seems
> > like it would scale better.
> >
> > -Jakob
> >
> > [1] https://blogs.apache.org/foundation/entry/success-at-apache-a-newbie
> >
> > On 15 September 2018 at 13:29, Jeff Payne <jpayne@bombora.com> wrote:
> > > How many operators are added to contrib per month? Is it too many to
> > make the decision case by case? If so, then the above mentioned rule
> sounds
> > fairly reasonable. However, if that's the rule, shouldn't a bunch of
> > existing modules be moved from contrib to core?
> > >
> > > Get Outlook for Android<https://aka.ms/ghei36>
> > >
> > > ________________________________
> > > From: Taylor Edmiston <tedmiston@gmail.com>
> > > Sent: Saturday, September 15, 2018 1:13:47 PM
> > > To: dev@airflow.incubator.apache.org
> > > Subject: Re: Guidelines on Contrib vs Non-contrib
> > >
> > > My understanding as a contributor is that if a hook/operator is in
> core,
> > it
> > > means that a committer is willing to take personal responsibility to
> > > maintain it (or at least help maintain it), and everything else goes in
> > > contrib.
> > >
> > > *Taylor Edmiston*
> > > Blog <https://blog.tedmiston.com/> | LinkedIn
> > > <https://www.linkedin.com/in/tedmiston/> | Stack Overflow
> > > <https://stackoverflow.com/users/149428/taylor-edmiston> | Developer
> > Story
> > > <https://stackoverflow.com/story/taylor>
> > >
> > >
> > >
> > > On Sat, Sep 15, 2018 at 2:02 PM Kaxil Naik <kaxilnaik@gmail.com>
> wrote:
> > >
> > >> Hi, all (mainly contributors),
> > >>
> > >> Can we decide on a common guideline on when a hook/operator should go
> > under
> > >> contrib vs core?
> > >>
> > >> Regards,
> > >>
> > >> *Kaxil Naik*
> > >> *Big Data Consultant *@ *Data Reply UK*
> > >> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark &
> > Neo4j
> > >> Developer
> > >> *Phone: *+44 (0) 74820 88992
> > >> *LinkedIn*: https://www.linkedin.com/in/kaxil
> > >>
> >
> --
> *  •  **Tim Swast*
> *  •  *Software Friendliness Engineer
> *  •  *Google Cloud Developer Relations
> *  •  *Seattle, WA, USA
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message