airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Felix Cheung (JIRA)" <>
Subject [jira] [Commented] (AIRFLOW-198) Create an ShortCircuitIfNotCurrentOperator
Date Thu, 18 Aug 2016 10:28:20 GMT


Felix Cheung commented on AIRFLOW-198:

I'm new but would love to look into this.
Sounds like we still need "Avoid scheduling multiple instances of a task that has been marked
as only_run_latest and prioritize the most recent execution date"

> Create an ShortCircuitIfNotCurrentOperator
> ------------------------------------------
>                 Key: AIRFLOW-198
>                 URL:
>             Project: Apache Airflow
>          Issue Type: New Feature
>            Reporter: Peter Attardo
>            Assignee: Siddharth Anand
> Taken from:
> "For cases where we need to only run the latest in a series of task instance runs and
mark the others as skipped. For example, we may have job to execute a DB snapshot every day.
If the DAG is paused for 5 days and then unpaused, we don’t want to run all 5, just the
latest. With this feature, we will provide “cron” functionality for task scheduling that
is not related to ETL"
> I've decided to implement this as a subclass of ShortCircuitOperator - OnlyRunCurrentOperator/ShortCircuitIfNotCurrentOperator
will skip downstream if the dag run being executed is not current.

This message was sent by Atlassian JIRA

View raw message