airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From drew.sonne@gmail.com <drew.so...@gmail.com>
Subject Re: Further improvements to plugin handling
Date Tue, 19 Feb 2019 07:49:57 GMT
I, unfortunatly, couldn't make it to the Meetup. What was discussed in regards to plugins?

On 2019/02/18 12:27:46, David Sabater Dinter <davidsabater@google.com.INVALID> wrote:

> Hi all,
> Just following up from this recommendation and the discussion we had during
> our last Airflow London Meetup @Google
> <https://www.meetup.com/London-Apache-Airflow-Meetup/events/257090138/>.
> Are there any plans to make that official in anyway?
> It would be great to have some direction/steer so the community can
> implement custom hooks/operators in the right way.
> 
> 
> Kind regards.
> 
> On Mon, 14 Jan 2019 at 13:50, Björn Pollex
> <bjoern.pollex@soundcloud.com.invalid> wrote:
> 
> > This is what we’ve been doing internally, and if it becomes the “official”
> > way of doing things, then I’m more than happy.
> >
> > Cheers,
> >
> >         Björn
> >
> > > On 14. Jan 2019, at 13:38, Ash Berlin-Taylor <ash@apache.org> wrote:
> > >
> > > Operators and hooks are just normal python modules, and don't actually
> > need to be defined under `airflow.` to work.
> > >
> > > My advice (and something I've been meaning to update the docs to say) is
> > to just make your custom hooks/ops a normal class and do
> > >
> > >    from myco.airflow.operators.my_op import MyOp
> > >
> > > So long as the "myco" package is installed on the scheduler and worker
> > nodes this will work fine (this is what we do at work)
> > >
> > > The plugin mechanism is I think still needed to get Flask
> > views/menu_items integrated (as the need to be registered with the app at
> > creation time)?
> > >
> > > So: not to namespace packages as I don't think we need it.
> > >
> > > -ash
> > >
> > >> On 14 Jan 2019, at 11:58, Björn Pollex <bjoern.pollex@soundcloud.com.INVALID>
> > wrote:
> > >>
> > >> With the recent (awesome) new feature to [load plugins from entry
> > points][1], I’m wondering if this could be taken further, to make handling
> > plugins simpler and less magic. One pain point for me personally has been
> > that, since plugins are loaded by Airflow magic, my IDE cannot give me any
> > help (e.g. syntax completion) when using operators, hooks or similar that
> > are loaded from plugins, because it can’t find the source files based on
> > the package name. This could be fixed by changing the packages `airflow`,
> > `airflow.hooks`, `airflow.operators` etc. into [namespace packages][2].
> > With such a change, one wouldn’t even need to declare a plugin in order to
> > distribute a new hook or operator, a plain Python package is enough.
> > >>
> > >> Furthermore, I wonder if we need the `AirflowPlugin` class at all.
> > Instead, we could have dedicated entry points for views, custom menus,
> > etc., and each package would register relevant classes with those entry
> > points directly. This would remove a lot of magic from Airflow, and instead
> > use established tools to handle the complexity.
> > >>
> > >> I believe it would be possible to implement both of these without
> > breaking existing code, so a migration path is available. The main problem
> > is that we would need to move all code that currently resides in the
> > `__init__.py` files of the prospective namespace packages elsewhere (the
> > constituent parts of namespace packages are loaded in a non-deterministic
> > order, so one can’t rely on code any specific `__init__.py` if a namespace
> > package to be executed first, or, in fact, at all), but as far as I can
> > see, everything in there can neatly fit somewhere else.
> > >>
> > >> I would love to start working on this, if there is a chance that at
> > least parts of this would find acceptance. What does the Airflow dev
> > community think about this?
> > >>
> > >> Cheers,
> > >>
> > >>      Björn Pollex
> > >>
> > >> [1]: https://github.com/apache/airflow/pull/4412
> > >> [2]: https://packaging.python.org/guides/packaging-namespace-packages/
> > >>
> > >
> >
> >
> 
> -- 
> 
> 
> David Sabater
> 
> Google Cloud | Cluster Customer Engineer
> 
> davidsabater@google.com
> 
> Belgrave House, 76 Buckingham Palace Road, London SW1W 9TQ
> <https://maps.google.com/?q=76+Buckingham+Palace+Road,+London+SW1W+9TQ&entry=gmail&source=g>
> 
> Tel +44 (0)7826 532190
> 
> 
> This email may be confidential and privileged. If you received this
> communication by mistake, please don't forward it to anyone else, please
> erase all copies and attachments, and please let me know that it has gone
> to the wrong person.
> 
> The above terms reflect a potential business arrangement, are provided
> solely as a basis for further discussion, and are not intended to be and do
> not constitute a legally binding obligation. No legally binding obligations
> will be created, implied, or inferred until an agreement in final form is
> executed in writing by all parties involved.
> 
> Google UK Limited
> Registered Office: Belgrave House, 76 Buckingham Palace Road, London SW1W
> 9TQ
> Registered in England Number: 3977902
> 

Mime
View raw message