airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ash Berlin-Taylor <...@apache.org>
Subject Re: Further improvements to plugin handling
Date Mon, 14 Jan 2019 12:38:20 GMT
Operators and hooks are just normal python modules, and don't actually need to be defined under
`airflow.` to work.

My advice (and something I've been meaning to update the docs to say) is to just make your
custom hooks/ops a normal class and do

    from myco.airflow.operators.my_op import MyOp

So long as the "myco" package is installed on the scheduler and worker nodes this will work
fine (this is what we do at work)

The plugin mechanism is I think still needed to get Flask views/menu_items integrated (as
the need to be registered with the app at creation time)?

So: not to namespace packages as I don't think we need it.

-ash

> On 14 Jan 2019, at 11:58, Björn Pollex <bjoern.pollex@soundcloud.com.INVALID>
wrote:
> 
> With the recent (awesome) new feature to [load plugins from entry points][1], I’m wondering
if this could be taken further, to make handling plugins simpler and less magic. One pain
point for me personally has been that, since plugins are loaded by Airflow magic, my IDE cannot
give me any help (e.g. syntax completion) when using operators, hooks or similar that are
loaded from plugins, because it can’t find the source files based on the package name. This
could be fixed by changing the packages `airflow`, `airflow.hooks`, `airflow.operators` etc.
into [namespace packages][2]. With such a change, one wouldn’t even need to declare a plugin
in order to distribute a new hook or operator, a plain Python package is enough.
> 
> Furthermore, I wonder if we need the `AirflowPlugin` class at all. Instead, we could
have dedicated entry points for views, custom menus, etc., and each package would register
relevant classes with those entry points directly. This would remove a lot of magic from Airflow,
and instead use established tools to handle the complexity.
> 
> I believe it would be possible to implement both of these without breaking existing code,
so a migration path is available. The main problem is that we would need to move all code
that currently resides in the `__init__.py` files of the prospective namespace packages elsewhere
(the constituent parts of namespace packages are loaded in a non-deterministic order, so one
can’t rely on code any specific `__init__.py` if a namespace package to be executed first,
or, in fact, at all), but as far as I can see, everything in there can neatly fit somewhere
else.
> 
> I would love to start working on this, if there is a chance that at least parts of this
would find acceptance. What does the Airflow dev community think about this?
> 
> Cheers,
> 
> 	Björn Pollex
> 
> [1]: https://github.com/apache/airflow/pull/4412
> [2]: https://packaging.python.org/guides/packaging-namespace-packages/
> 


Mime
View raw message