airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ash Berlin-Taylor <...@apache.org>
Subject Re: [2.0 spring cleaning] Deprecate adding Operators and Hooks via plugins?
Date Fri, 12 Apr 2019 15:53:25 GMT
Can you give an example?

We can still have custom hooks/operators per install, I'm just saying they don't need to be
implemented/"registered" with airflow.plugin -- all that provides is another way of having
that be imported.

-ash

> On 12 Apr 2019, at 16:21, Chen Tong <cixuuz@gmail.com> wrote:
> 
> IMHO, it's worth to have such dynamic hooks adding ability. I met issues
> before that the current hook cannot satisfy the needs. I have to write my
> own hook and hacked Connections. If hook is not coupled tightly with
> Connections, it will be easier by switching to another python package.
> 
> On Fri, Apr 12, 2019 at 11:14 AM Daniel Imberman <daniel.imberman@gmail.com>
> wrote:
> 
>> I am all for this. The only complexity is ensuring the operator/hook
>> exists in the workers too, but overall highly for.
>> On Apr 12, 2019, 7:37 AM -0700, James Meickle <jmeickle@quantopian.com.invalid>,
>> wrote:
>>> YES - I strongly agree with this! I first did it this way because I
>> wanted
>>> to follow the instructions, assuming there was some Airflow magic, and
>>> later found it really frustrating to maintain. We should be clear that
>>> standard Python packaging is the way to go.
>>> 
>>> That being said, what if Airflow had an official Cookiecutter template
>> for
>>> Airflow operator packages, and suggested that as a way to manage your
>>> operators in the docs? That would probably help people who aren't as
>>> familiar with Python, and over time might increase quality of third party
>>> operators (if it ships by default with linting/etc.)
>>> 
>>> On Fri, Apr 12, 2019 at 3:54 AM Ash Berlin-Taylor <ash@apache.org>
>> wrote:
>>> 
>>>> This is something I've said a number of times (but possibly never on
>> the
>>>> list):
>>>> 
>>>> I think we should deprecate adding Operators and Hooks via the Airflow
>>>> plugin mechanism.
>>>> 
>>>> I think plugins should be reserved for any mechanism that a plain-ol
>>>> python module import won't work for (which is basically anything that
>> needs
>>>> to tie deeply in to the Webserver or Scheduler process).
>>>> 
>>>> To that endI think we should deprecate adding operators via plugins:
>>>> 
>>>> from airflow.operators.my_plugin import MyOperator
>>>> 
>>>> can become
>>>> 
>>>> from my_plugin import MyOperator
>>>> 
>>>> with no impact on functionality.
>>>> 
>>>> The same could be said for hooks, with one possible feature addition:
>>>> Right now the list of connection types in the Connections screen is
>>>> hard-coded. Is it possible worth making that dynamic somehow based on
>> the
>>>> Hook classes that are loaded? i.e. adding the ability for hooks added
>> in
>>>> plguins to add items to that list?
>>>> 
>>>> -ash
>> 


Mime
View raw message