airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Felix Cheung <felixcheun...@hotmail.com>
Subject Re: ShortCircuitIfNotCurrentOperator
Date Thu, 18 Aug 2016 00:54:53 GMT
Apache Oozie has LAST_ONLY execution policy in which only the current instance will run when
catching up or backfilling.

https://oozie.apache.org/docs/4.2.0/CoordinatorFunctionalSpec.html#a6.3._Synchronous_Coordinator_Application_Definition


_____________________________
From: siddharth anand <sanand@apache.org<mailto:sanand@apache.org>>
Sent: Wednesday, August 17, 2016 5:39 PM
Subject: Re: ShortCircuitIfNotCurrentOperator
To: <dev@airflow.incubator.apache.org<mailto:dev@airflow.incubator.apache.org>>


Hmn... none that I know of, but it seems pretty useful for us!

On Wed, Aug 17, 2016 at 4:43 PM, Felix Cheung <felixcheung_m@hotmail.com<mailto:felixcheung_m@hotmail.com>>
wrote:

> Cool. Other scheduler has a concept called LAST_ONLY?
>
>
>
>
>
>
> On Wed, Aug 17, 2016 at 2:34 PM -0700, "siddharth anand" <
> sanand@apache.org<mailto:sanand@apache.org><mailto:sanand@apache.org>>
wrote:
>
> For the longest time, we have been discussing supporting full cron
> functionality in Airflow. Currently, Airflow supports cron syntax with one
> pretty visible issue.
>
> For example, if you wanted cron to schedule a job every day at midnight,
> you would specify 0 0 * * *. In Airflow, if you were to pause your dag for
> a week and then unpause it, your dag would run the past 7 days' worth of
> dag runs. However, a common use-case for cron schedules is to use it for
> some routine job scheduling (e.g. full db backup, log file cleanup,
> minutely alert condition checking) that only needs to run at midnight, but
> doesn't need to "backfill" the days missed.
>
> We had originally envisioned building this into the scheduler, but that is
> unnecessarily complex. Instead, we have had the ShortCircuitOperator around
> for a while. I plan to extend this to make a
> ShortCircuitIfNotCurrentOperator. This will skip downstream for dag runs
> that are not current.
>
> https://issues.apache.org/jira/browse/AIRFLOW-198
>
> FYI.. alternate names for this operator are welcome.
> -s
>



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message