airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxime Beauchemin <maximebeauche...@gmail.com>
Subject Re: What are the advantages of plugins, not sure I see any?
Date Fri, 30 Mar 2018 20:37:30 GMT
Yes, it makes most sense to just have a `common/hooks` and
`common/operators/` in the repo where your DAGs live and import them at
will.

Max

On Fri, Mar 30, 2018 at 1:30 PM, Kyle Hamlin <hamlin.kn@gmail.com> wrote:

> Thanks for the responses! I think my conclusion was similar, they seem good
> for redistribution, but if you're only working with operators and hooks and
> aren't sharing that code then it might not make too much sense to use them.
>
> On Fri, Mar 30, 2018 at 4:23 PM Maxime Beauchemin <
> maximebeauchemin@gmail.com> wrote:
>
> > The original intent was to use plugins as a way to share sets of objects
> > and applications build on top of Airflow.
> >
> > For instance it'd be possible to ship the things listed bellow as Airflow
> > plugin:
> >
> > * "validate-and-schedule my query" UI
> > * a set of ML-related hooks and operators that match a specific workflow
> > * a Hive metastore-browser UI
> > * drag and drop UI
> > * ...
> >
> > As far as I know it's not a super popular feature. Maybe the scope of
> > Airflow is already large enough without having all that stuff sitting on
> > top of it. :)
> >
> > As George pointed out it could also allow to accelerate the release
> cadence
> > of sets of things that are currently inside Airflow. Things like Google
> > Cloud-related operators and hooks could ship as a plugin on their own
> > release schedule.
> >
> > Max
> >
> > On Thu, Mar 29, 2018 at 11:07 PM, Alex Tronchin-James 949-412-7220
> > <(949)%20412-7220> <
> > alex.n.james@gmail.com> wrote:
> >
> > > At Netflix we've put our plugin inside the DAGs folder and pointed the
> > > config to it there so we can both import directly in DAGs AND update
> the
> > > plugin as we go. This makes it easy to test changes to operators needed
> > for
> > > ongoing DAG development in the same PR.
> > >
> > > The two plugin features I've used which don't translate to the direct
> > > import approach are custom macros (we provide some internal libraries)
> > and
> > > UI menu links, which we use for linking local docs describing our
> > > deployment and custom operators, server/worker monitoring with atlas,
> and
> > > genie job monitoring.
> > >
> > > On Thu, Mar 29, 2018 at 4:56 PM George Leslie-Waksman
> > > <george@cloverhealth.com.invalid> wrote:
> > >
> > > > It's presumably useful if you want to package your plugins for other
> > > people
> > > > to use but it seems like everyone just adds those directly to the
> > Airflow
> > > > codebase these days.
> > > >
> > > > On Thu, Mar 29, 2018 at 4:27 PM Kyle Hamlin <hamlin.kn@gmail.com>
> > wrote:
> > > >
> > > > > Yeah so far I have only written hooks and operators so maybe the
> > > benefit
> > > > > only  kicks in for other airflow abstractions.
> > > > >
> > > > > > On Mar 29, 2018, at 7:15 PM, George Leslie-Waksman <
> > > > > george@cloverhealth.com.INVALID> wrote:
> > > > > >
> > > > > > We also import our operators and sensors directly.
> > > > > >
> > > > > > However, executors and some other pieces are a little bit harder
> to
> > > > deal
> > > > > > with as non-plugins
> > > > > >
> > > > > >> On Thu, Mar 29, 2018 at 3:56 PM Kyle Hamlin <
> hamlin.kn@gmail.com>
> > > > > wrote:
> > > > > >>
> > > > > >> Hello,
> > > > > >>
> > > > > >> I just got done writing a few plugins, and the process has
left
> me
> > > > > >> wondering what the real benefits are? As far as I can tell,
it
> > makes
> > > > > >> testing more difficult since you cannot import from the
created
> > > > module,
> > > > > you
> > > > > >> have to import directly from the plugin. Additionally, your
code
> > > > editor
> > > > > >> isn't aware of these new plugin modules since they are created
> > when
> > > > you
> > > > > >> start the app up, this makes it seem like there are errors
when
> > > there
> > > > > >> aren't. Why not just create a lib/ dir with hooks, operators
> etc..
> > > > dirs
> > > > > >> inside and be done with it? Very curious what peoples thoughts
> > are,
> > > > who
> > > > > >> knows I could be testing wrong or writing the plugins wrong.
> > Thanks
> > > in
> > > > > >> advance!
> > > > > >>
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message