airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Conrad Lee <con...@parsely.com>
Subject [AIRFLOW-1428] Skipped tasks deadlock dag runs
Date Wed, 19 Jul 2017 11:31:03 GMT
I've run into issues in dag runs where skipped tasks cause the dag to
deadlock.  I think the deeper issue here is an airflow community issue:
there has been an unresolved debate around which trigger rule should
propagate skipped tasks.

If you're interested in the problem, check out AIRFLOW-1428
<https://issues.apache.org/jira/browse/AIRFLOW-1428>. I've

   - provided a minimal example
   - dug through the source code exploration for the root cause, and
   - documented bits of the community debate

I don't want to write up a PR until we've somehow decided on the desired
semantics/behavior-- is there a core committer who could own the decision
making process here?

*How things currently stand*
There seems to be support for adding a successs_or_skipped trigger rule,
but nobody has implemented it.  I would be happy to implement this--I've
scoped out the changes necessary and the addition of this trigger rule
would not be such a large project.

The bigger question seems to be what would happen to the (default!)
all_success trigger rule in the presence of an all_success_or_skipped
trigger rule.  Would this trigger rule always fail if any upstream tasks
were skipped?

Thanks,
Conrad

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message