airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Martin Grayson (JIRA)" <>
Subject [jira] [Commented] (AIRFLOW-564) DagRunOperator and ExternalTaskSensor Incompatibility
Date Tue, 11 Oct 2016 14:58:21 GMT


Martin Grayson commented on AIRFLOW-564:

Thanks for the inspiration [~lauralorenz]! It didn't quite occur to me that I could pass the
execution date of the parent dag into the child (via the payload) and subsequently use that
in the ExternalTaskSensor.

This could be a workaround for me, but does complicate the situation. I agree with you that
inheritance of execution_date could be a more optimal solution.

> DagRunOperator and ExternalTaskSensor Incompatibility
> -----------------------------------------------------
>                 Key: AIRFLOW-564
>                 URL:
>             Project: Apache Airflow
>          Issue Type: Task
>          Components: DagRun
>    Affects Versions: Airflow
>            Reporter: Martin Grayson
>         Attachments: 1.controller.png, 2. par.png, 3. dep.png,,,
> I have an hourly Dag that acts to orchestrate other dags, triggering dag runs if a precondition
is met.
> The controller dag (Dag CD) uses a `TriggerDagRunOperator` operator to do this, launching
2 other dags (Dag A, Dag B) in my example (10+ in my real life). 
> Within these dags they have their own dependencies, for example Dag B is dependent on
Dag A's completion. This is handled using a `ExternalTaskSensor`. 
> Since Dag A and B are executed dynamically with the `TriggerDagRunOperator`, they're
assigned very granular execution dates, rather than dd-mm-yyyy hh:00:00 as the scheduler would
assign.  Due to this, the `ExternalTaskSensor` will not succeed unless Dag A and B managed
to have exactly the same execution_date (down to the second).
> As far as I can tell, I'm not able to force the `ExternalTaskSensor` to poke for the
correct execution date. The only possible solution would be to implement a `execution_date_fn`
that looked through the metadata db to find the exact execution time. This seems massively
complicated for a relatively simple requirement.
> Can this scenario be modelled with these components?
> I've attached a set of example dags and some screenshots, showing the issue.
> Thanks,
> Martin

This message was sent by Atlassian JIRA

View raw message