airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raphael Lopez Kaufman (JIRA)" <>
Subject [jira] [Commented] (AIRFLOW-1414) Add support for retriggering dependent workflows
Date Mon, 17 Jul 2017 09:28:00 GMT


Raphael Lopez Kaufman commented on AIRFLOW-1414:

[~bolke] Any though on that this (the corresponding PR is
We are trying to switch from Oozie to Airflow at and would really need this feature
(or maybe there's already a way to achieve this) for us to go forward with the migration.

> Add support for retriggering dependent workflows
> ------------------------------------------------
>                 Key: AIRFLOW-1414
>                 URL:
>             Project: Apache Airflow
>          Issue Type: Improvement
>            Reporter: Raphael Lopez Kaufman
> Currently when using an ExternalTaskSensor sensor, to have a dag wait for the completion
of a task in another dag, there is no way, when clearing the task that is depended on, to
also clear the ExternalTaskSensor task (and its downstream/upstream tasks).
> However, that might be important when teams have separate responsibilities and create
multi-staged data pipelines. Let's say there is a team responsible for transforming logs to
make them available in hadoop land. They maintain a hourly dag processing the raw logs. Other
teams wait on the completion of this dag using an ExternalTaskSensor. If the logging team
realizes that some logs where corrupted, not only do they have to rerun the task that processed
these logs but also all the dags that use these logs to generate higher level data.
> Introducing the notion of descendants can solve this problem. The descendants of dag
A are the dags that have an ExternalTaskSensor pointing to a task of dag A. Then, when clearing
a task in the view, an option can be added to also clear the tasks of the descendants (having
the upstream, downstream, recursive, future and past flags act accordingly).

This message was sent by Atlassian JIRA

View raw message