airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tao Feng <fengta...@gmail.com>
Subject Re: [2.0 spring cleaning] Deprecate subdags
Date Fri, 12 Apr 2019 17:50:11 GMT
+1 on fixing the subdags.

I think one issue I recall is that subdag uses backfill scheduler which
doesn't have the on same functionality as normal scheduler. E.g the tasks
within subdag are not resource bound by the pool. But with Chao-han's fix
on https://github.com/apache/airflow/pull/4949, I think the issue should be
fixed.

tldr, per my understanding, the underneath issue is due to backfill
scheduler needs to have more love :)

But we should fix / improve the subdag as it has been used by many
different companies.

On Fri, Apr 12, 2019 at 10:37 AM Bolke de Bruin <bdbruin@gmail.com> wrote:

> +1
>
> Sub dags should be fixed within the scheduler and run normally.
>
>
>
>
> On 12 April 2019 at 19:36:27, Feng Lu (fenglu@google.com.invalid) wrote:
>
> Agree with others who think SubDag should stay, we should fix the SubDag
> implementation but not remove the abstraction itself.
>
> On Fri, Apr 12, 2019 at 8:42 AM Chen Tong <cixuuz@gmail.com> wrote:
>
> > Is it possible to re-implement it in the view-level, not in operator
> level?
> > And this operator is just define a different view in GUI, that these
> tasks
> > will be collapsed into another view.
> >
> > On Fri, Apr 12, 2019 at 11:31 AM James Meickle
> > <jmeickle@quantopian.com.invalid> wrote:
> >
> > > I have avoided using them because of outstanding issues like the open
> > JIRA
> > > issues I linked above, or similar issues that I've read about on blog
> > > posts. If it were just GUI or UX issues I'd use them, but many people
> > have
> > > reported issues which affect concurrency/stability, consistency, or
> > > correctness of results. I believe that it's working for you, but for
> me,
> > > it's not worth the risk to build using them in my environment (even
> > though
> > > they could be handy for many of our workflows).
> > >
> > > On Fri, Apr 12, 2019 at 11:18 AM Kaxil Naik <kaxilnaik@gmail.com>
> wrote:
> > >
> > > > I have been using SubDags in production and haven't had much problem
> > with
> > > > it.
> > > >
> > > > Can you list the issues you had?
> > > >
> > > > Regards,
> > > > Kaxil
> > > >
> > > >
> > > > On Fri, Apr 12, 2019, 16:16 James Meickle <jmeickle@quantopian.com
> > > > .invalid>
> > > > wrote:
> > > >
> > > > > Given their bad reputation, would it be appropriate to deprecate
> > > subDAGs
> > > > > now to advertise that they're no longer considered a suitable
> > > > > implementation? If a new and better implementation is created,
> would
> > it
> > > > > even be similar enough to subDAGs that we'd want to continue to
> call
> > > the
> > > > > feature that?
> > > > >
> > > > > They feel like a "new Airflow user trap" right now - I have had to
> > tell
> > > > my
> > > > > team never to use them, because they seem appealing and are in the
> > > > official
> > > > > docs.
> > > > >
> > > > > On Fri, Apr 12, 2019 at 10:51 AM Ash Berlin-Taylor <ash@apache.org
> >
> > > > wrote:
> > > > >
> > > > > > I'd like to find time to fix subdags as they do provide a useful
> > > > > > abstraction - but I agree right now they aren't great (I avoid
> them
> > > > > because
> > > > > > of this)
> > > > > >
> > > > > > I have half thoughts of how to it should work, I just need to
> look
> > at
> > > > the
> > > > > > code in depth to see if that makes sense. Now 1.10.3 is out
I
> might
> > > > have
> > > > > a
> > > > > > bit more time to do this.
> > > > > >
> > > > > > -ash
> > > > > >
> > > > > > > On 12 Apr 2019, at 15:48, James Meickle <
> jmeickle@quantopian.com
> > > > > .INVALID>
> > > > > > wrote:
> > > > > > >
> > > > > > > I think we should deprecate SubDAGs given the complexity
they
> add
> > > and
> > > > > the
> > > > > > > limited usage and use cases. Or, we should invest effort
in
> > > > redesigning
> > > > > > > their API and implementation. I think that having to account
> for
> > > > > > > subdag-introduced complexity makes Airflow's code much
harder
> to
> > > > > maintain
> > > > > > > and buggier, looking at how many open issues there are
that
> > > reference
> > > > > > > subdags (and how unrelated in topic they are):
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> https://issues.apache.org/jira/browse/AIRFLOW-3292?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Open%20AND%20text%20~%20%22subdag%22
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message