airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Björn Pollex <bjoern.pol...@soundcloud.com.INVALID>
Subject Re: Further improvements to plugin handling
Date Mon, 14 Jan 2019 13:50:04 GMT
This is what we’ve been doing internally, and if it becomes the “official” way of doing
things, then I’m more than happy.

Cheers,

	Björn

> On 14. Jan 2019, at 13:38, Ash Berlin-Taylor <ash@apache.org> wrote:
> 
> Operators and hooks are just normal python modules, and don't actually need to be defined
under `airflow.` to work.
> 
> My advice (and something I've been meaning to update the docs to say) is to just make
your custom hooks/ops a normal class and do
> 
>    from myco.airflow.operators.my_op import MyOp
> 
> So long as the "myco" package is installed on the scheduler and worker nodes this will
work fine (this is what we do at work)
> 
> The plugin mechanism is I think still needed to get Flask views/menu_items integrated
(as the need to be registered with the app at creation time)?
> 
> So: not to namespace packages as I don't think we need it.
> 
> -ash
> 
>> On 14 Jan 2019, at 11:58, Björn Pollex <bjoern.pollex@soundcloud.com.INVALID>
wrote:
>> 
>> With the recent (awesome) new feature to [load plugins from entry points][1], I’m
wondering if this could be taken further, to make handling plugins simpler and less magic.
One pain point for me personally has been that, since plugins are loaded by Airflow magic,
my IDE cannot give me any help (e.g. syntax completion) when using operators, hooks or similar
that are loaded from plugins, because it can’t find the source files based on the package
name. This could be fixed by changing the packages `airflow`, `airflow.hooks`, `airflow.operators`
etc. into [namespace packages][2]. With such a change, one wouldn’t even need to declare
a plugin in order to distribute a new hook or operator, a plain Python package is enough.
>> 
>> Furthermore, I wonder if we need the `AirflowPlugin` class at all. Instead, we could
have dedicated entry points for views, custom menus, etc., and each package would register
relevant classes with those entry points directly. This would remove a lot of magic from Airflow,
and instead use established tools to handle the complexity.
>> 
>> I believe it would be possible to implement both of these without breaking existing
code, so a migration path is available. The main problem is that we would need to move all
code that currently resides in the `__init__.py` files of the prospective namespace packages
elsewhere (the constituent parts of namespace packages are loaded in a non-deterministic order,
so one can’t rely on code any specific `__init__.py` if a namespace package to be executed
first, or, in fact, at all), but as far as I can see, everything in there can neatly fit somewhere
else.
>> 
>> I would love to start working on this, if there is a chance that at least parts of
this would find acceptance. What does the Airflow dev community think about this?
>> 
>> Cheers,
>> 
>> 	Björn Pollex
>> 
>> [1]: https://github.com/apache/airflow/pull/4412
>> [2]: https://packaging.python.org/guides/packaging-namespace-packages/
>> 
> 


Mime
View raw message