airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Sabater Dinter <davidsaba...@google.com.INVALID>
Subject Re: Further improvements to plugin handling
Date Mon, 18 Feb 2019 12:27:46 GMT
Hi all,
Just following up from this recommendation and the discussion we had during
our last Airflow London Meetup @Google
<https://www.meetup.com/London-Apache-Airflow-Meetup/events/257090138/>.
Are there any plans to make that official in anyway?
It would be great to have some direction/steer so the community can
implement custom hooks/operators in the right way.


Kind regards.

On Mon, 14 Jan 2019 at 13:50, Björn Pollex
<bjoern.pollex@soundcloud.com.invalid> wrote:

> This is what we’ve been doing internally, and if it becomes the “official”
> way of doing things, then I’m more than happy.
>
> Cheers,
>
>         Björn
>
> > On 14. Jan 2019, at 13:38, Ash Berlin-Taylor <ash@apache.org> wrote:
> >
> > Operators and hooks are just normal python modules, and don't actually
> need to be defined under `airflow.` to work.
> >
> > My advice (and something I've been meaning to update the docs to say) is
> to just make your custom hooks/ops a normal class and do
> >
> >    from myco.airflow.operators.my_op import MyOp
> >
> > So long as the "myco" package is installed on the scheduler and worker
> nodes this will work fine (this is what we do at work)
> >
> > The plugin mechanism is I think still needed to get Flask
> views/menu_items integrated (as the need to be registered with the app at
> creation time)?
> >
> > So: not to namespace packages as I don't think we need it.
> >
> > -ash
> >
> >> On 14 Jan 2019, at 11:58, Björn Pollex <bjoern.pollex@soundcloud.com.INVALID>
> wrote:
> >>
> >> With the recent (awesome) new feature to [load plugins from entry
> points][1], I’m wondering if this could be taken further, to make handling
> plugins simpler and less magic. One pain point for me personally has been
> that, since plugins are loaded by Airflow magic, my IDE cannot give me any
> help (e.g. syntax completion) when using operators, hooks or similar that
> are loaded from plugins, because it can’t find the source files based on
> the package name. This could be fixed by changing the packages `airflow`,
> `airflow.hooks`, `airflow.operators` etc. into [namespace packages][2].
> With such a change, one wouldn’t even need to declare a plugin in order to
> distribute a new hook or operator, a plain Python package is enough.
> >>
> >> Furthermore, I wonder if we need the `AirflowPlugin` class at all.
> Instead, we could have dedicated entry points for views, custom menus,
> etc., and each package would register relevant classes with those entry
> points directly. This would remove a lot of magic from Airflow, and instead
> use established tools to handle the complexity.
> >>
> >> I believe it would be possible to implement both of these without
> breaking existing code, so a migration path is available. The main problem
> is that we would need to move all code that currently resides in the
> `__init__.py` files of the prospective namespace packages elsewhere (the
> constituent parts of namespace packages are loaded in a non-deterministic
> order, so one can’t rely on code any specific `__init__.py` if a namespace
> package to be executed first, or, in fact, at all), but as far as I can
> see, everything in there can neatly fit somewhere else.
> >>
> >> I would love to start working on this, if there is a chance that at
> least parts of this would find acceptance. What does the Airflow dev
> community think about this?
> >>
> >> Cheers,
> >>
> >>      Björn Pollex
> >>
> >> [1]: https://github.com/apache/airflow/pull/4412
> >> [2]: https://packaging.python.org/guides/packaging-namespace-packages/
> >>
> >
>
>

-- 


David Sabater

Google Cloud | Cluster Customer Engineer

davidsabater@google.com

Belgrave House, 76 Buckingham Palace Road, London SW1W 9TQ
<https://maps.google.com/?q=76+Buckingham+Palace+Road,+London+SW1W+9TQ&entry=gmail&source=g>

Tel +44 (0)7826 532190


This email may be confidential and privileged. If you received this
communication by mistake, please don't forward it to anyone else, please
erase all copies and attachments, and please let me know that it has gone
to the wrong person.

The above terms reflect a potential business arrangement, are provided
solely as a basis for further discussion, and are not intended to be and do
not constitute a legally binding obligation. No legally binding obligations
will be created, implied, or inferred until an agreement in final form is
executed in writing by all parties involved.

Google UK Limited
Registered Office: Belgrave House, 76 Buckingham Palace Road, London SW1W
9TQ
Registered in England Number: 3977902

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message