airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ash Berlin-Taylor <...@apache.org>
Subject Re: [2.0 spring cleaning] Deprecate subdags
Date Fri, 12 Apr 2019 19:06:06 GMT
This is what I was thinking - the dag collector in the scheduler should "just" be able to collect
the tasks for subdags up to the parent dag. I'd possibly go as far as saying no DagRun object
for subdags too.

(Yes, "just" will never be that simple).

-a

On 12 April 2019 18:37:24 BST, Bolke de Bruin <bdbruin@gmail.com> wrote:
>+1
>
>Sub dags should be fixed within the scheduler and run normally.
>
>
>
>
>On 12 April 2019 at 19:36:27, Feng Lu (fenglu@google.com.invalid)
>wrote:
>
>Agree with others who think SubDag should stay, we should fix the
>SubDag
>implementation but not remove the abstraction itself.
>
>On Fri, Apr 12, 2019 at 8:42 AM Chen Tong <cixuuz@gmail.com> wrote:
>
>> Is it possible to re-implement it in the view-level, not in operator
>level?
>> And this operator is just define a different view in GUI, that these
>tasks
>> will be collapsed into another view.
>>
>> On Fri, Apr 12, 2019 at 11:31 AM James Meickle
>> <jmeickle@quantopian.com.invalid> wrote:
>>
>> > I have avoided using them because of outstanding issues like the
>open
>> JIRA
>> > issues I linked above, or similar issues that I've read about on
>blog
>> > posts. If it were just GUI or UX issues I'd use them, but many
>people
>> have
>> > reported issues which affect concurrency/stability, consistency, or
>> > correctness of results. I believe that it's working for you, but
>for
>me,
>> > it's not worth the risk to build using them in my environment (even
>> though
>> > they could be handy for many of our workflows).
>> >
>> > On Fri, Apr 12, 2019 at 11:18 AM Kaxil Naik <kaxilnaik@gmail.com>
>wrote:
>> >
>> > > I have been using SubDags in production and haven't had much
>problem
>> with
>> > > it.
>> > >
>> > > Can you list the issues you had?
>> > >
>> > > Regards,
>> > > Kaxil
>> > >
>> > >
>> > > On Fri, Apr 12, 2019, 16:16 James Meickle
><jmeickle@quantopian.com
>> > > .invalid>
>> > > wrote:
>> > >
>> > > > Given their bad reputation, would it be appropriate to
>deprecate
>> > subDAGs
>> > > > now to advertise that they're no longer considered a suitable
>> > > > implementation? If a new and better implementation is created,
>would
>> it
>> > > > even be similar enough to subDAGs that we'd want to continue to
>call
>> > the
>> > > > feature that?
>> > > >
>> > > > They feel like a "new Airflow user trap" right now - I have had
>to
>> tell
>> > > my
>> > > > team never to use them, because they seem appealing and are in
>the
>> > > official
>> > > > docs.
>> > > >
>> > > > On Fri, Apr 12, 2019 at 10:51 AM Ash Berlin-Taylor
><ash@apache.org>
>> > > wrote:
>> > > >
>> > > > > I'd like to find time to fix subdags as they do provide a
>useful
>> > > > > abstraction - but I agree right now they aren't great (I
>avoid
>them
>> > > > because
>> > > > > of this)
>> > > > >
>> > > > > I have half thoughts of how to it should work, I just need to
>look
>> at
>> > > the
>> > > > > code in depth to see if that makes sense. Now 1.10.3 is out I
>might
>> > > have
>> > > > a
>> > > > > bit more time to do this.
>> > > > >
>> > > > > -ash
>> > > > >
>> > > > > > On 12 Apr 2019, at 15:48, James Meickle
><jmeickle@quantopian.com
>> > > > .INVALID>
>> > > > > wrote:
>> > > > > >
>> > > > > > I think we should deprecate SubDAGs given the complexity
>they
>add
>> > and
>> > > > the
>> > > > > > limited usage and use cases. Or, we should invest effort
in
>> > > redesigning
>> > > > > > their API and implementation. I think that having to
>account
>for
>> > > > > > subdag-introduced complexity makes Airflow's code much
>harder
>to
>> > > > maintain
>> > > > > > and buggier, looking at how many open issues there are that
>> > reference
>> > > > > > subdags (and how unrelated in topic they are):
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>https://issues.apache.org/jira/browse/AIRFLOW-3292?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Open%20AND%20text%20~%20%22subdag%22
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>>

Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message