airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chen Tong <cix...@gmail.com>
Subject Re: [2.0 spring cleaning] Deprecate subdags
Date Fri, 12 Apr 2019 15:42:12 GMT
Is it possible to re-implement it in the view-level, not in operator level?
And this operator is just define a different view in GUI, that these tasks
will be collapsed into another view.

On Fri, Apr 12, 2019 at 11:31 AM James Meickle
<jmeickle@quantopian.com.invalid> wrote:

> I have avoided using them because of outstanding issues like the open JIRA
> issues I linked above, or similar issues that I've read about on blog
> posts. If it were just GUI or UX issues I'd use them, but many people have
> reported issues which affect concurrency/stability, consistency, or
> correctness of results. I believe that it's working for you, but for me,
> it's not worth the risk to build using them in my environment (even though
> they could be handy for many of our workflows).
>
> On Fri, Apr 12, 2019 at 11:18 AM Kaxil Naik <kaxilnaik@gmail.com> wrote:
>
> > I have been using SubDags in production and haven't had much problem with
> > it.
> >
> > Can you list the issues you had?
> >
> > Regards,
> > Kaxil
> >
> >
> > On Fri, Apr 12, 2019, 16:16 James Meickle <jmeickle@quantopian.com
> > .invalid>
> > wrote:
> >
> > > Given their bad reputation, would it be appropriate to deprecate
> subDAGs
> > > now to advertise that they're no longer considered a suitable
> > > implementation? If a new and better implementation is created, would it
> > > even be similar enough to subDAGs that we'd want to continue to call
> the
> > > feature that?
> > >
> > > They feel like a "new Airflow user trap" right now - I have had to tell
> > my
> > > team never to use them, because they seem appealing and are in the
> > official
> > > docs.
> > >
> > > On Fri, Apr 12, 2019 at 10:51 AM Ash Berlin-Taylor <ash@apache.org>
> > wrote:
> > >
> > > > I'd like to find time to fix subdags as they do provide a useful
> > > > abstraction - but I agree right now they aren't great (I avoid them
> > > because
> > > > of this)
> > > >
> > > > I have half thoughts of how to it should work, I just need to look at
> > the
> > > > code in depth to see if that makes sense. Now 1.10.3 is out I might
> > have
> > > a
> > > > bit more time to do this.
> > > >
> > > > -ash
> > > >
> > > > > On 12 Apr 2019, at 15:48, James Meickle <jmeickle@quantopian.com
> > > .INVALID>
> > > > wrote:
> > > > >
> > > > > I think we should deprecate SubDAGs given the complexity they add
> and
> > > the
> > > > > limited usage and use cases. Or, we should invest effort in
> > redesigning
> > > > > their API and implementation. I think that having to account for
> > > > > subdag-introduced complexity makes Airflow's code much harder to
> > > maintain
> > > > > and buggier, looking at how many open issues there are that
> reference
> > > > > subdags (and how unrelated in topic they are):
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/AIRFLOW-3292?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Open%20AND%20text%20~%20%22subdag%22
> > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message