airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: [2.0 spring cleaning] Deprecate subdags
Date Fri, 12 Apr 2019 17:37:24 GMT
+1

Sub dags should be fixed within the scheduler and run normally.




On 12 April 2019 at 19:36:27, Feng Lu (fenglu@google.com.invalid) wrote:

Agree with others who think SubDag should stay, we should fix the SubDag
implementation but not remove the abstraction itself.

On Fri, Apr 12, 2019 at 8:42 AM Chen Tong <cixuuz@gmail.com> wrote:

> Is it possible to re-implement it in the view-level, not in operator
level?
> And this operator is just define a different view in GUI, that these
tasks
> will be collapsed into another view.
>
> On Fri, Apr 12, 2019 at 11:31 AM James Meickle
> <jmeickle@quantopian.com.invalid> wrote:
>
> > I have avoided using them because of outstanding issues like the open
> JIRA
> > issues I linked above, or similar issues that I've read about on blog
> > posts. If it were just GUI or UX issues I'd use them, but many people
> have
> > reported issues which affect concurrency/stability, consistency, or
> > correctness of results. I believe that it's working for you, but for
me,
> > it's not worth the risk to build using them in my environment (even
> though
> > they could be handy for many of our workflows).
> >
> > On Fri, Apr 12, 2019 at 11:18 AM Kaxil Naik <kaxilnaik@gmail.com>
wrote:
> >
> > > I have been using SubDags in production and haven't had much problem
> with
> > > it.
> > >
> > > Can you list the issues you had?
> > >
> > > Regards,
> > > Kaxil
> > >
> > >
> > > On Fri, Apr 12, 2019, 16:16 James Meickle <jmeickle@quantopian.com
> > > .invalid>
> > > wrote:
> > >
> > > > Given their bad reputation, would it be appropriate to deprecate
> > subDAGs
> > > > now to advertise that they're no longer considered a suitable
> > > > implementation? If a new and better implementation is created,
would
> it
> > > > even be similar enough to subDAGs that we'd want to continue to
call
> > the
> > > > feature that?
> > > >
> > > > They feel like a "new Airflow user trap" right now - I have had to
> tell
> > > my
> > > > team never to use them, because they seem appealing and are in the
> > > official
> > > > docs.
> > > >
> > > > On Fri, Apr 12, 2019 at 10:51 AM Ash Berlin-Taylor <ash@apache.org>
> > > wrote:
> > > >
> > > > > I'd like to find time to fix subdags as they do provide a useful
> > > > > abstraction - but I agree right now they aren't great (I avoid
them
> > > > because
> > > > > of this)
> > > > >
> > > > > I have half thoughts of how to it should work, I just need to
look
> at
> > > the
> > > > > code in depth to see if that makes sense. Now 1.10.3 is out I
might
> > > have
> > > > a
> > > > > bit more time to do this.
> > > > >
> > > > > -ash
> > > > >
> > > > > > On 12 Apr 2019, at 15:48, James Meickle <jmeickle@quantopian.com
> > > > .INVALID>
> > > > > wrote:
> > > > > >
> > > > > > I think we should deprecate SubDAGs given the complexity they
add
> > and
> > > > the
> > > > > > limited usage and use cases. Or, we should invest effort in
> > > redesigning
> > > > > > their API and implementation. I think that having to account
for
> > > > > > subdag-introduced complexity makes Airflow's code much harder
to
> > > > maintain
> > > > > > and buggier, looking at how many open issues there are that
> > reference
> > > > > > subdags (and how unrelated in topic they are):
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://issues.apache.org/jira/browse/AIRFLOW-3292?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Open%20AND%20text%20~%20%22subdag%22
> > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message