airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ash Berlin-Taylor <...@apache.org>
Subject Re: [2.0 spring cleaning] Deprecate contrib folder?
Date Fri, 12 Apr 2019 15:17:30 GMT
Somewhat off topic from the topic of just moving the well tested parts out of contrib/ but
okay. I thought I had somewhere but I can't find where I did it before.

Quickly:

- How do we go about testing changes to core against hook?
- Who/what/when releases the split-out hooks. Are they ASF releases or community. (If community
then this can happen right now. Someone can just create a package, publish on pypi and we
can start pointing at that)
- What about CI? How do we test changes to core against operators? When? How often?
- Some hooks need tighter integration (GCP,S3, Azure, ElasticSearch) as they can be used by
loggers. How do we do that?
- A "operator" package should probably depend upon Airflow to get "BaseOperator" class etc,
but that means that installing the `apache-airflow[extra]` can't work as that would be a loop.
Maybe we need an `apache-airflow-core` instead.

In short: I like "mono" repo as it has lots of advantages :)

-ash


> On 12 Apr 2019, at 16:10, Chen Tong <cixuuz@gmail.com> wrote:
> 
> Hi Ash,
> 
> Could you state the questions you are concerned about? We may discuss it in
> this thread.
> 
> Thank you!
> 
> On Fri, Apr 12, 2019 at 11:06 AM Ash Berlin-Taylor <ash@apache.org> wrote:
> 
>> Speaking as a release manager AIP-8 also had some large unanswered
>> workflow questions before I'd be happy about to "accept" it.
>> 
>> -ash
>> 
>>> On 12 Apr 2019, at 16:04, Chen Tong <cixuuz@gmail.com> wrote:
>>> 
>>> AIP-8 is intended to achieve this by separate operators/hooks to
>> individual
>>> packages. But the author hasn't been actively working on this recently.
>>> 
>>> On Fri, Apr 12, 2019 at 9:29 AM Julian De Ruiter <
>>> julianderuiter@godatadriven.com> wrote:
>>> 
>>>> Isn’t this in contradiction with AIP-8, which is aimed at removing
>>>> operators/hooks from the core Airflow package?
>>>> 
>>>> Personally I would rather remove hooks/operators from Airflow than add
>>>> even more to the Airflow core. This counts double for the contrib stuff,
>>>> which is often poorly designed and/or tested.
>>>> 
>>>> Best,
>>>> Julian
>>>> 
>>>>> On 12 Apr 2019, at 10:23, Bolke de Bruin <bdbruin@gmail.com> wrote:
>>>>> 
>>>>> That’s perfectly fine to me.
>>>>> 
>>>>> Verstuurd vanaf mijn iPad
>>>>> 
>>>>>> Op 12 apr. 2019 om 10:20 heeft Kaxil Naik <kaxilnaik@gmail.com>
het
>>>> volgende geschreven:
>>>>>> 
>>>>>> Ok. How about moving the properly tested and maintained hooks/ops
from
>>>>>> contrib to core?
>>>>>> 
>>>>>>> On Fri, Apr 12, 2019, 09:13 Bolke de Bruin <bdbruin@gmail.com>
>> wrote:
>>>>>>> 
>>>>>>> I disagree. Core signals “properly tested” and maintained.
Ie. A kind
>>>> of
>>>>>>> quality.  I don’t think contrib has that.
>>>>>>> 
>>>>>>> Verstuurd vanaf mijn iPad
>>>>>>> 
>>>>>>>> Op 12 apr. 2019 om 10:03 heeft Kaxil Naik <kaxilnaik@gmail.com>
het
>>>>>>> volgende geschreven:
>>>>>>>> 
>>>>>>>> Contrib folder was used when it was used at Airbnb. Currently,
it
>>>> doesn't
>>>>>>>> make any sense and we have equal responsibility to maintain
all the
>>>>>>> hooks,
>>>>>>>> operators, sensors in contrib folder as we do for core.
>>>>>>>> 
>>>>>>>> I would suggest to remove contrib folder and move all hooks,
ops,
>> and
>>>>>>>> sensors to the core folder.
>>>>>>>> 
>>>>>>>> Or reorganize the folder structure similar to what was discussed
in
>> a
>>>>>>>> mailing thread few months ago.
>>>>>>>> 
>>>>>>>> Regards,
>>>>>>>> Kaxil
>>>>>>> 
>>>> 
>>>> 
>> 
>> 


Mime
View raw message