airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dan Davydov (JIRA)" <>
Subject [jira] [Comment Edited] (AIRFLOW-992) Skipped tasks do not propagate correctly
Date Sat, 18 Mar 2017 00:39:41 GMT


Dan Davydov edited comment on AIRFLOW-992 at 3/18/17 12:39 AM:

Good point I'll do both edit + comment in the future, haven't tested it yet, we are going
to release the latest rc on Monday and then I'll be out for a bit but when I'm back I will
try this fix

was (Author: aoen):
Good point I'll do both edit + comment in the future

> Skipped tasks do not propagate correctly
> ----------------------------------------
>                 Key: AIRFLOW-992
>                 URL:
>             Project: Apache Airflow
>          Issue Type: Bug
>            Reporter: Dan Davydov
>            Priority: Critical
> We are seeing skipped tasks not being propagated correctly:
> E.g. 
> A->B
> `-->C
> Task A depends on task B and C
> If B gets skipped and C gets run then:
> Expected:
> A will get skipped
> EDIT: Upon further investigation this was caused by a change in the semantics of ALL_SUCCESS,
which I have these feelings about:
> Intuitively you would expect to skip any task that has dependencies that weren't run
by default, i.e. the trigger rule is called ALL_SUCCESS and skipped tasks are not successful
ones, and that was also the old behavior in 1.7.3.
> This is going to break some use cases which could be alright, but I feel these new semantics
make less sense than before so it's a bad reason to break existing use cases.
> I will get started on a PR for a new ALL_SUCCESS_NOT_SKIPPED trigger rule but again I
feel this is hacky and really we should have the old ALL_SUCCESS (default) and a new ALL_SUCCESS_OR_SKIPPED
trigger rule if desired.
> Actual:
> A gets run
> It's possible the reason for the change in semantics (which was to fix a bug) is fixed
already by and the only thing we need
to do is revert
> [~bolke]

This message was sent by Atlassian JIRA

View raw message