airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yati (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AIRFLOW-1488) Add a sensor operator to wait on DagRuns
Date Sun, 06 Aug 2017 12:16:03 GMT
Yati created AIRFLOW-1488:
-----------------------------

             Summary: Add a sensor operator to wait on DagRuns
                 Key: AIRFLOW-1488
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1488
             Project: Apache Airflow
          Issue Type: New Feature
          Components: contrib, operators
            Reporter: Yati
            Assignee: Yati


The [ExternalTaskSensor|https://airflow.incubator.apache.org/code.html#airflow.operators.ExternalTaskSensor]
operator already allows for encoding dependencies on tasks in external DAGs. However, when
you have teams, each owning multiple small-to-medium sized DAGs, it is desirable to be able
to wait on an external DagRun as a whole. This allows the owners of an upstream DAG to refactor
their code freely by splitting/squashing task responsibilities, without worrying about dependent
DAGs breaking.

I'll now enumerate the easiest ways of achieving this that come to mind:

* Make all DAGs always have a join DummyOperator in the end, with a task id that follows some
convention, e.g., "{{ dag_id }}.__end__".
* Make ExternalTaskSensor poke for a DagRun instead of TaskInstances when the external_task_id
argument is None.
* Implement a separate DagRunSensor operator.

After considerations, we decided to implement a separate operator, which we've been using
in the team for our workflows, and I think it would make a good addition to contrib.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message