airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kaxil Naik <kaxiln...@gmail.com>
Subject Re: [2.0 spring cleaning] Deprecate subdags
Date Fri, 12 Apr 2019 15:18:23 GMT
I have been using SubDags in production and haven't had much problem with
it.

Can you list the issues you had?

Regards,
Kaxil


On Fri, Apr 12, 2019, 16:16 James Meickle <jmeickle@quantopian.com.invalid>
wrote:

> Given their bad reputation, would it be appropriate to deprecate subDAGs
> now to advertise that they're no longer considered a suitable
> implementation? If a new and better implementation is created, would it
> even be similar enough to subDAGs that we'd want to continue to call the
> feature that?
>
> They feel like a "new Airflow user trap" right now - I have had to tell my
> team never to use them, because they seem appealing and are in the official
> docs.
>
> On Fri, Apr 12, 2019 at 10:51 AM Ash Berlin-Taylor <ash@apache.org> wrote:
>
> > I'd like to find time to fix subdags as they do provide a useful
> > abstraction - but I agree right now they aren't great (I avoid them
> because
> > of this)
> >
> > I have half thoughts of how to it should work, I just need to look at the
> > code in depth to see if that makes sense. Now 1.10.3 is out I might have
> a
> > bit more time to do this.
> >
> > -ash
> >
> > > On 12 Apr 2019, at 15:48, James Meickle <jmeickle@quantopian.com
> .INVALID>
> > wrote:
> > >
> > > I think we should deprecate SubDAGs given the complexity they add and
> the
> > > limited usage and use cases. Or, we should invest effort in redesigning
> > > their API and implementation. I think that having to account for
> > > subdag-introduced complexity makes Airflow's code much harder to
> maintain
> > > and buggier, looking at how many open issues there are that reference
> > > subdags (and how unrelated in topic they are):
> > >
> >
> https://issues.apache.org/jira/browse/AIRFLOW-3292?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Open%20AND%20text%20~%20%22subdag%22
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message