airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Harmon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-2680) Don't automatically percolate skipped state
Date Thu, 15 Nov 2018 18:22:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-2680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16688469#comment-16688469
] 

Andrew Harmon commented on AIRFLOW-2680:
----------------------------------------

I'm looking for a solution to the same issue. I don't think you always want the SKIP status
to propagate downstream. It would be nice to have some more control of this

> Don't automatically percolate skipped state
> -------------------------------------------
>
>                 Key: AIRFLOW-2680
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2680
>             Project: Apache Airflow
>          Issue Type: Improvement
>            Reporter: Andrei-Alin Popescu
>            Assignee: Andrei-Alin Popescu
>            Priority: Major
>
> Dear Airflow Maintainers,
>  
> As part of our workflow, we have cases where all the upstream of a certain task A can
be skipped. In this case, airflow seems to automatically mark A as skipped.
> However, this does not quite work for us, since there are changes external to the DAG
which A needs to process, regardless of whether its upstream ran or not. Additionally, we
require A to get into an "upstream_failed" state and not run if any its upstream tasks failed.
> I don't see a trigger rule to cover this, so what would be the best way to achieve this?
I was thinking we could attach a DummyOperator as an upstream to A, which in a way marks the
fact that A depends on some external data and needs to run anyway, but this can get really
ugly for big DAGs.
> I was also thinking we could have a new trigger_rule, e.g. "no_failure", which would
only trigger tasks if no upstream has failed. It differs from "all_success" in that it will
also trigger if all upstream has been skipped, rather than percolating the skipped state on.
> I'd really appreciate your feedback on this, and I'd like to know if in fact there is
already a good way of doing this with airflow that I don't know of.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message