airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Tronchin-James 949-412-7220 <alex.n.ja...@gmail.com>
Subject Re: What are the advantages of plugins, not sure I see any?
Date Fri, 30 Mar 2018 06:07:01 GMT
At Netflix we've put our plugin inside the DAGs folder and pointed the
config to it there so we can both import directly in DAGs AND update the
plugin as we go. This makes it easy to test changes to operators needed for
ongoing DAG development in the same PR.

The two plugin features I've used which don't translate to the direct
import approach are custom macros (we provide some internal libraries) and
UI menu links, which we use for linking local docs describing our
deployment and custom operators, server/worker monitoring with atlas, and
genie job monitoring.

On Thu, Mar 29, 2018 at 4:56 PM George Leslie-Waksman
<george@cloverhealth.com.invalid> wrote:

> It's presumably useful if you want to package your plugins for other people
> to use but it seems like everyone just adds those directly to the Airflow
> codebase these days.
>
> On Thu, Mar 29, 2018 at 4:27 PM Kyle Hamlin <hamlin.kn@gmail.com> wrote:
>
> > Yeah so far I have only written hooks and operators so maybe the benefit
> > only  kicks in for other airflow abstractions.
> >
> > > On Mar 29, 2018, at 7:15 PM, George Leslie-Waksman <
> > george@cloverhealth.com.INVALID> wrote:
> > >
> > > We also import our operators and sensors directly.
> > >
> > > However, executors and some other pieces are a little bit harder to
> deal
> > > with as non-plugins
> > >
> > >> On Thu, Mar 29, 2018 at 3:56 PM Kyle Hamlin <hamlin.kn@gmail.com>
> > wrote:
> > >>
> > >> Hello,
> > >>
> > >> I just got done writing a few plugins, and the process has left me
> > >> wondering what the real benefits are? As far as I can tell, it makes
> > >> testing more difficult since you cannot import from the created
> module,
> > you
> > >> have to import directly from the plugin. Additionally, your code
> editor
> > >> isn't aware of these new plugin modules since they are created when
> you
> > >> start the app up, this makes it seem like there are errors when there
> > >> aren't. Why not just create a lib/ dir with hooks, operators etc..
> dirs
> > >> inside and be done with it? Very curious what peoples thoughts are,
> who
> > >> knows I could be testing wrong or writing the plugins wrong. Thanks in
> > >> advance!
> > >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message