airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ash Berlin-Taylor <...@apache.org>
Subject Re: Guidelines on Contrib vs Non-contrib
Date Tue, 18 Sep 2018 09:11:51 GMT
Operators and hooks don't need any special plugin system - simply having them as as separate
Python modules which are imported using normal python semantics is enough.

In fact now that I think about it: I want to deprecate the plugins registering hooks/operators
etc and limit it to only bits which a simple python import can't manage - which I think is
only anything that needs to be registered with another system, such as custom routes in the
web UI.

I'll draft an AIP for this soon.

-ash


> On 18 Sep 2018, at 00:50, George Leslie-Waksman <waksman@gmail.com> wrote:
> 
> Given we have a plugin system, could we alternatively move away from
> keeping non-core supported code outside of the core project/repo?
> 
> It would hugely decrease the surface area of the main repository and
> testing infrastructure to get most of the contrib code out to its own place.
> 
> Further, it would decrease the committer burden of having to approve/merge
> code that is not supposed to be their responsibility.
> 
> On Mon, Sep 17, 2018 at 4:37 PM Tim Swast <swast@google.com.invalid> wrote:
> 
>>> Individual operators and hooks living in separate repositories on github
>> (or possibly other Apache projects), which are then distributed by pip and
>> installed as libraries seems like it would scale better.
>> 
>> Pandas did this about a year ago, and it's seemed to have worked well. For
>> example, pandas.read_gbq is a very thin wrapper around pandas_gbq.read_gbq
>> (distributed as a separate package). It has made it easier for me to track
>> issues corresponding to my area of expertise.
>> 
>> On Sun, Sep 16, 2018 at 1:25 PM Jakob Homan <jghoman@gmail.com> wrote:
>> 
>>>> My understanding as a contributor is that if a hook/operator is in
>> core,
>>> it
>>>> means that a committer is willing to take personal responsibility to
>>>> maintain it (or at least help maintain it), and everything else goes in
>>>> contrib.
>>> 
>>> That's not correct.  All of the code is owned by the entire
>>> community[1]; no one person is responsible for any of it.  There's no
>>> silos, fiefdoms, walled gardens, etc.  If the community cannot support
>>> a piece of code it should be deprecated and subsequently removed.
>>> 
>>> Contrib sections are almost always problematic for this reason.
>>> Hadoop ended up abandoning its.  Because Airflow acts as a gathering
>>> point for so many disparate technologies (databases, storage systems,
>>> compute engines, etc.), trying to keep all of them corralled and up to
>>> date will be very difficult.  Individual operators and hooks living in
>>> separate repositories on github (or possibly other Apache projects),
>>> which are then distributed by pip and installed as libraries seems
>>> like it would scale better.
>>> 
>>> -Jakob
>>> 
>>> [1] https://blogs.apache.org/foundation/entry/success-at-apache-a-newbie
>>> 
>>> On 15 September 2018 at 13:29, Jeff Payne <jpayne@bombora.com> wrote:
>>>> How many operators are added to contrib per month? Is it too many to
>>> make the decision case by case? If so, then the above mentioned rule
>> sounds
>>> fairly reasonable. However, if that's the rule, shouldn't a bunch of
>>> existing modules be moved from contrib to core?
>>>> 
>>>> Get Outlook for Android<https://aka.ms/ghei36>
>>>> 
>>>> ________________________________
>>>> From: Taylor Edmiston <tedmiston@gmail.com>
>>>> Sent: Saturday, September 15, 2018 1:13:47 PM
>>>> To: dev@airflow.incubator.apache.org
>>>> Subject: Re: Guidelines on Contrib vs Non-contrib
>>>> 
>>>> My understanding as a contributor is that if a hook/operator is in
>> core,
>>> it
>>>> means that a committer is willing to take personal responsibility to
>>>> maintain it (or at least help maintain it), and everything else goes in
>>>> contrib.
>>>> 
>>>> *Taylor Edmiston*
>>>> Blog <https://blog.tedmiston.com/> | LinkedIn
>>>> <https://www.linkedin.com/in/tedmiston/> | Stack Overflow
>>>> <https://stackoverflow.com/users/149428/taylor-edmiston> | Developer
>>> Story
>>>> <https://stackoverflow.com/story/taylor>
>>>> 
>>>> 
>>>> 
>>>> On Sat, Sep 15, 2018 at 2:02 PM Kaxil Naik <kaxilnaik@gmail.com>
>> wrote:
>>>> 
>>>>> Hi, all (mainly contributors),
>>>>> 
>>>>> Can we decide on a common guideline on when a hook/operator should go
>>> under
>>>>> contrib vs core?
>>>>> 
>>>>> Regards,
>>>>> 
>>>>> *Kaxil Naik*
>>>>> *Big Data Consultant *@ *Data Reply UK*
>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark &
>>> Neo4j
>>>>> Developer
>>>>> *Phone: *+44 (0) 74820 88992
>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
>>>>> 
>>> 
>> --
>> *  •  **Tim Swast*
>> *  •  *Software Friendliness Engineer
>> *  •  *Google Cloud Developer Relations
>> *  •  *Seattle, WA, USA
>> 


Mime
View raw message