airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeremiah Lowin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-93) Allow specifying multiple task execution deltas for ExternalTaskSensors
Date Wed, 18 May 2016 17:08:13 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-93?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289340#comment-15289340
] 

Jeremiah Lowin commented on AIRFLOW-93:
---------------------------------------

[~underyx] I'm rapidly learning about ExternalTaskSensor but could you not create 6 ETS in
your hourly DAG, one for each of the 10 minute tasks (so one with delta of 10 minutes, one
with 20 minutes, one with 30 minutes, etc.) and then have your hourly task depend on all of
those sensors completing successfully? See for example AIRFLOW-47.

> Allow specifying multiple task execution deltas for ExternalTaskSensors
> -----------------------------------------------------------------------
>
>                 Key: AIRFLOW-93
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-93
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: operators
>    Affects Versions: Airflow 1.7.0
>            Reporter: Bence Nagy
>            Priority: Minor
>
> I have some {{ExternalTaskSensor}}s with a schedule interval of 1 hour, where the task
depended upon has a schedule interval of 10 minutes. Right now I'm depending only on the HH:50
execution, but it would be nice if I could specify a range that I need all executions from
HH:00 to HH:50 successful; otherwise if the depended upon tasks are executed out of order
the sensor will pass even though I don't have data for the earlier parts of the hour yet.
> A workaround would be to have one sensor for each 10 minutes of the hour, but that's
too nasty for me. Especially if my sensor's schedule interval would be 1 day.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message