airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Felix Cheung <felixcheun...@hotmail.com>
Subject Re: ShortCircuitIfNotCurrentOperator
Date Wed, 17 Aug 2016 23:43:26 GMT
Cool. Other scheduler has a concept called LAST_ONLY?






On Wed, Aug 17, 2016 at 2:34 PM -0700, "siddharth anand" <sanand@apache.org<mailto:sanand@apache.org>>
wrote:

For the longest time, we have been discussing supporting full cron
functionality in Airflow. Currently, Airflow supports cron syntax with one
pretty visible issue.

For example, if you wanted cron to schedule a job every day at midnight,
you would specify 0 0 * * *. In Airflow, if you were to pause your dag for
a week and then unpause it, your dag would run the past 7 days' worth of
dag runs. However, a common use-case for cron schedules is to use it for
some routine job scheduling (e.g. full db backup, log file cleanup,
minutely alert condition checking) that only needs to run at midnight, but
doesn't need to "backfill" the days missed.

We had originally envisioned building this into the scheduler, but that is
unnecessarily complex. Instead, we have had the ShortCircuitOperator around
for a while. I plan to extend this to make a
ShortCircuitIfNotCurrentOperator. This will skip downstream for dag runs
that are not current.

https://issues.apache.org/jira/browse/AIRFLOW-198

FYI.. alternate names for this operator are welcome.
-s

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message