airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeremiah Lowin (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AIRFLOW-100) Add flexibility to ExternalTaskSensor
Date Tue, 10 May 2016 23:34:12 GMT
Jeremiah Lowin created AIRFLOW-100:
--------------------------------------

             Summary: Add flexibility to ExternalTaskSensor
                 Key: AIRFLOW-100
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-100
             Project: Apache Airflow
          Issue Type: Improvement
          Components: operators
            Reporter: Jeremiah Lowin
            Assignee: Jeremiah Lowin
            Priority: Minor


The ExternalTaskSensor defaults to sensing tasks with the same {{execution_date}} as it does,
and has an {{execution_delta}} parameter for looking back farther in time. However, this doesn't
support the case where the sensing task has a smaller schedule_interval than the target task.

For example, if the ETS were run every hour, one couldn't use a fixed {{execution_delta}}
to sense a task that only ran daily (since each instance of the ETS would need a different
execution_delta). 

However, a Daily task can wait for multiple hourly tasks, because it knows in advance that
it needs 24 ETS's with deltas == range(24).

Concrete suggestion:
- add a param ({{execution_delta_fn}}?) that takes in the current execution_date and is expected
to return the desired sense date (for example, it could always return midnight of the previous
day, no matter what the ETS was executed).

cc [~criccomini]




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message