airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Victor Noagbodji <vnoagbo...@amplify-analytics.com>
Subject Re: Guidelines on Contrib vs Non-contrib
Date Tue, 18 Sep 2018 16:25:52 GMT
yes, please!

> On Sep 18, 2018, at 12:23 PM, Maxime Beauchemin <maximebeauchemin@gmail.com> wrote:
> 
> +1 for deprecating operators/hooks as plugins, let's use Python's good old
> python packages and maybe python "entry points" if we want to inject them
> in "airflow.operators"/"airflow.hooks" (which is probably not necessary)
> 
> On Tue, Sep 18, 2018 at 2:12 AM Ash Berlin-Taylor <ash@apache.org> wrote:
> 
>> Operators and hooks don't need any special plugin system - simply having
>> them as as separate Python modules which are imported using normal python
>> semantics is enough.
>> 
>> In fact now that I think about it: I want to deprecate the plugins
>> registering hooks/operators etc and limit it to only bits which a simple
>> python import can't manage - which I think is only anything that needs to
>> be registered with another system, such as custom routes in the web UI.
>> 
>> I'll draft an AIP for this soon.
>> 
>> -ash
>> 
>> 
>>> On 18 Sep 2018, at 00:50, George Leslie-Waksman <waksman@gmail.com>
>> wrote:
>>> 
>>> Given we have a plugin system, could we alternatively move away from
>>> keeping non-core supported code outside of the core project/repo?
>>> 
>>> It would hugely decrease the surface area of the main repository and
>>> testing infrastructure to get most of the contrib code out to its own
>> place.
>>> 
>>> Further, it would decrease the committer burden of having to
>> approve/merge
>>> code that is not supposed to be their responsibility.
>>> 
>>> On Mon, Sep 17, 2018 at 4:37 PM Tim Swast <swast@google.com.invalid>
>> wrote:
>>> 
>>>>> Individual operators and hooks living in separate repositories on
>> github
>>>> (or possibly other Apache projects), which are then distributed by pip
>> and
>>>> installed as libraries seems like it would scale better.
>>>> 
>>>> Pandas did this about a year ago, and it's seemed to have worked well.
>> For
>>>> example, pandas.read_gbq is a very thin wrapper around
>> pandas_gbq.read_gbq
>>>> (distributed as a separate package). It has made it easier for me to
>> track
>>>> issues corresponding to my area of expertise.
>>>> 
>>>> On Sun, Sep 16, 2018 at 1:25 PM Jakob Homan <jghoman@gmail.com> wrote:
>>>> 
>>>>>> My understanding as a contributor is that if a hook/operator is in
>>>> core,
>>>>> it
>>>>>> means that a committer is willing to take personal responsibility
to
>>>>>> maintain it (or at least help maintain it), and everything else goes
>> in
>>>>>> contrib.
>>>>> 
>>>>> That's not correct.  All of the code is owned by the entire
>>>>> community[1]; no one person is responsible for any of it.  There's no
>>>>> silos, fiefdoms, walled gardens, etc.  If the community cannot support
>>>>> a piece of code it should be deprecated and subsequently removed.
>>>>> 
>>>>> Contrib sections are almost always problematic for this reason.
>>>>> Hadoop ended up abandoning its.  Because Airflow acts as a gathering
>>>>> point for so many disparate technologies (databases, storage systems,
>>>>> compute engines, etc.), trying to keep all of them corralled and up to
>>>>> date will be very difficult.  Individual operators and hooks living in
>>>>> separate repositories on github (or possibly other Apache projects),
>>>>> which are then distributed by pip and installed as libraries seems
>>>>> like it would scale better.
>>>>> 
>>>>> -Jakob
>>>>> 
>>>>> [1]
>> https://blogs.apache.org/foundation/entry/success-at-apache-a-newbie
>>>>> 
>>>>> On 15 September 2018 at 13:29, Jeff Payne <jpayne@bombora.com>
wrote:
>>>>>> How many operators are added to contrib per month? Is it too many
to
>>>>> make the decision case by case? If so, then the above mentioned rule
>>>> sounds
>>>>> fairly reasonable. However, if that's the rule, shouldn't a bunch of
>>>>> existing modules be moved from contrib to core?
>>>>>> 
>>>>>> Get Outlook for Android<https://aka.ms/ghei36>
>>>>>> 
>>>>>> ________________________________
>>>>>> From: Taylor Edmiston <tedmiston@gmail.com>
>>>>>> Sent: Saturday, September 15, 2018 1:13:47 PM
>>>>>> To: dev@airflow.incubator.apache.org
>>>>>> Subject: Re: Guidelines on Contrib vs Non-contrib
>>>>>> 
>>>>>> My understanding as a contributor is that if a hook/operator is in
>>>> core,
>>>>> it
>>>>>> means that a committer is willing to take personal responsibility
to
>>>>>> maintain it (or at least help maintain it), and everything else goes
>> in
>>>>>> contrib.
>>>>>> 
>>>>>> *Taylor Edmiston*
>>>>>> Blog <https://blog.tedmiston.com/> | LinkedIn
>>>>>> <https://www.linkedin.com/in/tedmiston/> | Stack Overflow
>>>>>> <https://stackoverflow.com/users/149428/taylor-edmiston> |
Developer
>>>>> Story
>>>>>> <https://stackoverflow.com/story/taylor>
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Sat, Sep 15, 2018 at 2:02 PM Kaxil Naik <kaxilnaik@gmail.com>
>>>> wrote:
>>>>>> 
>>>>>>> Hi, all (mainly contributors),
>>>>>>> 
>>>>>>> Can we decide on a common guideline on when a hook/operator should
go
>>>>> under
>>>>>>> contrib vs core?
>>>>>>> 
>>>>>>> Regards,
>>>>>>> 
>>>>>>> *Kaxil Naik*
>>>>>>> *Big Data Consultant *@ *Data Reply UK*
>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark
&
>>>>> Neo4j
>>>>>>> Developer
>>>>>>> *Phone: *+44 (0) 74820 88992
>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil
>>>>>>> 
>>>>> 
>>>> --
>>>> *  •  **Tim Swast*
>>>> *  •  *Software Friendliness Engineer
>>>> *  •  *Google Cloud Developer Relations
>>>> *  •  *Seattle, WA, USA
>>>> 
>> 
>> 

Mime
View raw message