airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From siddharth anand <san...@apache.org>
Subject Re: ShortCircuitIfNotCurrentOperator
Date Thu, 18 Aug 2016 00:39:13 GMT
Hmn... none that I know of, but it seems pretty useful for us!

On Wed, Aug 17, 2016 at 4:43 PM, Felix Cheung <felixcheung_m@hotmail.com>
wrote:

> Cool. Other scheduler has a concept called LAST_ONLY?
>
>
>
>
>
>
> On Wed, Aug 17, 2016 at 2:34 PM -0700, "siddharth anand" <
> sanand@apache.org<mailto:sanand@apache.org>> wrote:
>
> For the longest time, we have been discussing supporting full cron
> functionality in Airflow. Currently, Airflow supports cron syntax with one
> pretty visible issue.
>
> For example, if you wanted cron to schedule a job every day at midnight,
> you would specify 0 0 * * *. In Airflow, if you were to pause your dag for
> a week and then unpause it, your dag would run the past 7 days' worth of
> dag runs. However, a common use-case for cron schedules is to use it for
> some routine job scheduling (e.g. full db backup, log file cleanup,
> minutely alert condition checking) that only needs to run at midnight, but
> doesn't need to "backfill" the days missed.
>
> We had originally envisioned building this into the scheduler, but that is
> unnecessarily complex. Instead, we have had the ShortCircuitOperator around
> for a while. I plan to extend this to make a
> ShortCircuitIfNotCurrentOperator. This will skip downstream for dag runs
> that are not current.
>
> https://issues.apache.org/jira/browse/AIRFLOW-198
>
> FYI.. alternate names for this operator are welcome.
> -s
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message