Mailing-List: contact commits-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
Date: Sun, 6 Aug 2017 12:16:03 +0000 (UTC)
From: "Yati (JIRA)" <jira@apache.org>
To: commits@airflow.incubator.apache.org
Message-ID: <JIRA.13092722.1502021741000.102194.1502021763061@Atlassian.JIRA>
In-Reply-To: <JIRA.13092722.1502021741000@Atlassian.JIRA>
References: <JIRA.13092722.1502021741000@Atlassian.JIRA> <JIRA.13092722.1502021741466@jira-lw-us.apache.org>
Subject: [jira] [Created] (AIRFLOW-1488) Add a sensor operator to wait on
 DagRuns
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Sun, 06 Aug 2017 12:16:08 -0000

Yati created AIRFLOW-1488:
-----------------------------

             Summary: Add a sensor operator to wait on DagRuns
                 Key: AIRFLOW-1488
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1488
             Project: Apache Airflow
          Issue Type: New Feature
          Components: contrib, operators
            Reporter: Yati
            Assignee: Yati


The [ExternalTaskSensor|https://airflow.incubator.apache.org/code.html#airflow.operators.ExternalTaskSensor] operator already allows for encoding dependencies on tasks in external DAGs. However, when you have teams, each owning multiple small-to-medium sized DAGs, it is desirable to be able to wait on an external DagRun as a whole. This allows the owners of an upstream DAG to refactor their code freely by splitting/squashing task responsibilities, without worrying about dependent DAGs breaking.

I'll now enumerate the easiest ways of achieving this that come to mind:

* Make all DAGs always have a join DummyOperator in the end, with a task id that follows some convention, e.g., "{{ dag_id }}.__end__".
* Make ExternalTaskSensor poke for a DagRun instead of TaskInstances when the external_task_id argument is None.
* Implement a separate DagRunSensor operator.

After considerations, we decided to implement a separate operator, which we've been using in the team for our workflows, and I think it would make a good addition to contrib.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)