airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Björn Pollex <bjoern.pol...@soundcloud.com.INVALID>
Subject Further improvements to plugin handling
Date Mon, 14 Jan 2019 11:58:55 GMT
With the recent (awesome) new feature to [load plugins from entry points][1], I’m wondering
if this could be taken further, to make handling plugins simpler and less magic. One pain
point for me personally has been that, since plugins are loaded by Airflow magic, my IDE cannot
give me any help (e.g. syntax completion) when using operators, hooks or similar that are
loaded from plugins, because it can’t find the source files based on the package name. This
could be fixed by changing the packages `airflow`, `airflow.hooks`, `airflow.operators` etc.
into [namespace packages][2]. With such a change, one wouldn’t even need to declare a plugin
in order to distribute a new hook or operator, a plain Python package is enough.

Furthermore, I wonder if we need the `AirflowPlugin` class at all. Instead, we could have
dedicated entry points for views, custom menus, etc., and each package would register relevant
classes with those entry points directly. This would remove a lot of magic from Airflow, and
instead use established tools to handle the complexity.

I believe it would be possible to implement both of these without breaking existing code,
so a migration path is available. The main problem is that we would need to move all code
that currently resides in the `__init__.py` files of the prospective namespace packages elsewhere
(the constituent parts of namespace packages are loaded in a non-deterministic order, so one
can’t rely on code any specific `__init__.py` if a namespace package to be executed first,
or, in fact, at all), but as far as I can see, everything in there can neatly fit somewhere
else.

I would love to start working on this, if there is a chance that at least parts of this would
find acceptance. What does the Airflow dev community think about this?

Cheers,

	Björn Pollex

[1]: https://github.com/apache/airflow/pull/4412
[2]: https://packaging.python.org/guides/packaging-namespace-packages/


Mime
View raw message