airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shaw, Damian P. " <damian.sha...@credit-suisse.com>
Subject DAG "Schedule Filter Callback"?
Date Fri, 30 Aug 2019 17:17:00 GMT
Hi all,

After discussion at the NY Meetup this week I've been pondering how Airflow could support
custom schedules with very little change to core Airflow logic and keeping backwards compatibility.

As I understand the common way to support custom schedules is through a BranchOperator. You
provide logic that on a good date executes the "run" branch and on another date runs the "don't
run" branch which usually is a dummy operator.

There are 2 problems associated with it which would be useful to me (and I think the rest
of the community) to solve:

1.       depends_on_past does not play well with branching, because the "run" branch tasks
get marked as "skipped"

2.       Template variables like "prev_ds" and "next_ds" represent the underlying schedule
and not the actual schedule you are working on

I therefore propose a "schedule_filter_callback", a function which you provide at DAG creation
time that takes in some arguments (execution date, timezone, DAG?), and returns a Truthy or
Falsy result based on if this is a good date to execute on. If schedule_filter_callback is
None then the current schedule logic is applied.

I appreciate this is a fairly significant proposal, but it seems like because it would just
be 1 extra argument on the DAG and make no change to the default behavior it doesn't quite
rise to the level of AIP? Sorry if this has already been discussed before.

Regards,
Damian


=============================================================================== 
Please access the attached hyperlink for an important electronic communications disclaimer:

http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html 
=============================================================================== 

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message