airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kaxil Naik <kaxiln...@gmail.com>
Subject Re: Guidelines on Contrib vs Non-contrib
Date Tue, 18 Sep 2018 08:39:23 GMT
+1 for moving contribs into each of their separate repos, we have
been talking about these for several months now. This should go into
Airflow 2.0. I will draft an AIP (Airflow Improvement Proposal) with some
proposed steps at the end of this week. If anyone wants to draft it before
that, feel free to do it.

On Tue, Sep 18, 2018 at 7:15 AM Jeff Payne <jpayne@bombora.com> wrote:

> George, did you mean "move towards keeping non-core supported code
> outside of the core project/repo?"
>
>
> [image: 1500660321366_bombora_logo-org-small] <http://bombora.com/>
> jeff payne | technical lead & senior data engineer
> 257 Park Ave S | 6th Floor | New York, NY 10010
> m: +1 916-273-0938| e: jpayne@bombora.com| @bomboradata
> <https://twitter.com/bomboradata>
>
> *Get our free report on using Intent Data for content and Account-Based
> Marketing*
> <http://bombora.com/powering-content-marketing-intent-data/?utm_source=email&utm_medium=signature&utm_campaign=contentreport>
> [image: 1500660332982_2017]
>
> ------------------------------
> *From:* George Leslie-Waksman <waksman@gmail.com>
> *Sent:* Monday, September 17, 2018 4:50:39 PM
> *To:* dev@airflow.incubator.apache.org
> *Subject:* Re: Guidelines on Contrib vs Non-contrib
>
> Given we have a plugin system, could we alternatively move away from
> keeping non-core supported code outside of the core project/repo?
>
> It would hugely decrease the surface area of the main repository and
> testing infrastructure to get most of the contrib code out to its own
> place.
>
> Further, it would decrease the committer burden of having to approve/merge
> code that is not supposed to be their responsibility.
>
> On Mon, Sep 17, 2018 at 4:37 PM Tim Swast <swast@google.com.invalid>
> wrote:
>
> > > Individual operators and hooks living in separate repositories on
> github
> > (or possibly other Apache projects), which are then distributed by pip
> and
> > installed as libraries seems like it would scale better.
> >
> > Pandas did this about a year ago, and it's seemed to have worked well.
> For
> > example, pandas.read_gbq is a very thin wrapper around
> pandas_gbq.read_gbq
> > (distributed as a separate package). It has made it easier for me to
> track
> > issues corresponding to my area of expertise.
> >
> > On Sun, Sep 16, 2018 at 1:25 PM Jakob Homan <jghoman@gmail.com> wrote:
> >
> > > > My understanding as a contributor is that if a hook/operator is in
> > core,
> > > it
> > > > means that a committer is willing to take personal responsibility to
> > > > maintain it (or at least help maintain it), and everything else goes
> in
> > > > contrib.
> > >
> > > That's not correct.  All of the code is owned by the entire
> > > community[1]; no one person is responsible for any of it.  There's no
> > > silos, fiefdoms, walled gardens, etc.  If the community cannot support
> > > a piece of code it should be deprecated and subsequently removed.
> > >
> > > Contrib sections are almost always problematic for this reason.
> > > Hadoop ended up abandoning its.  Because Airflow acts as a gathering
> > > point for so many disparate technologies (databases, storage systems,
> > > compute engines, etc.), trying to keep all of them corralled and up to
> > > date will be very difficult.  Individual operators and hooks living in
> > > separate repositories on github (or possibly other Apache projects),
> > > which are then distributed by pip and installed as libraries seems
> > > like it would scale better.
> > >
> > > -Jakob
> > >
> > > [1]
> https://blogs.apache.org/foundation/entry/success-at-apache-a-newbie
> > >
> > > On 15 September 2018 at 13:29, Jeff Payne <jpayne@bombora.com> wrote:
> > > > How many operators are added to contrib per month? Is it too many to
> > > make the decision case by case? If so, then the above mentioned rule
> > sounds
> > > fairly reasonable. However, if that's the rule, shouldn't a bunch of
> > > existing modules be moved from contrib to core?
> > > >
> > > > Get Outlook for Android<https://aka.ms/ghei36>
> > > >
> > > > ________________________________
> > > > From: Taylor Edmiston <tedmiston@gmail.com>
> > > > Sent: Saturday, September 15, 2018 1:13:47 PM
> > > > To: dev@airflow.incubator.apache.org
> > > > Subject: Re: Guidelines on Contrib vs Non-contrib
> > > >
> > > > My understanding as a contributor is that if a hook/operator is in
> > core,
> > > it
> > > > means that a committer is willing to take personal responsibility to
> > > > maintain it (or at least help maintain it), and everything else goes
> in
> > > > contrib.
> > > >
> > > > *Taylor Edmiston*
> > > > Blog <https://blog.tedmiston.com/> | LinkedIn
> > > > <https://www.linkedin.com/in/tedmiston/> | Stack Overflow
> > > > <https://stackoverflow.com/users/149428/taylor-edmiston> | Developer
> > > Story
> > > > <https://stackoverflow.com/story/taylor>
> > > >
> > > >
> > > >
> > > > On Sat, Sep 15, 2018 at 2:02 PM Kaxil Naik <kaxilnaik@gmail.com>
> > wrote:
> > > >
> > > >> Hi, all (mainly contributors),
> > > >>
> > > >> Can we decide on a common guideline on when a hook/operator should
> go
> > > under
> > > >> contrib vs core?
> > > >>
> > > >> Regards,
> > > >>
> > > >> *Kaxil Naik*
> > > >> *Big Data Consultant *@ *Data Reply UK*
> > > >> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark
&
> > > Neo4j
> > > >> Developer
> > > >> *Phone: *+44 (0) 74820 88992
> > > >> *LinkedIn*: https://www.linkedin.com/in/kaxil
> > > >>
> > >
> > --
> > *  •  **Tim Swast*
> > *  •  *Software Friendliness Engineer
> > *  •  *Google Cloud Developer Relations
> > *  •  *Seattle, WA, USA
> >
>


-- 
*Kaxil Naik*
*Big Data Consultant *@ *Data Reply UK*
*Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
Developer
*Phone: *+44 (0) 74820 88992
*LinkedIn*: https://www.linkedin.com/in/kaxil

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message