airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andy Cooper (JIRA)" <>
Subject [jira] [Assigned] (AIRFLOW-81) Scheduler blackout time period
Date Sun, 05 Aug 2018 03:28:00 GMT


Andy Cooper reassigned AIRFLOW-81:

    Assignee: Andy Cooper

> Scheduler blackout time period
> ------------------------------
>                 Key: AIRFLOW-81
>                 URL:
>             Project: Apache Airflow
>          Issue Type: Wish
>          Components: scheduler
>            Reporter: Sean McIntyre
>            Assignee: Andy Cooper
>            Priority: Minor
>              Labels: features
> I have the need for a scheduler blackout time period in Airflow.
> My team, which uses Airflow, has been asked to not query one of my company's data sources
between midnight and 7 AM. When we launch big backfills on this data source, it would be nice
to have the Scheduler not schedule some TaskInstances during the blackout hours.
> We (@r39132 and @ledsusop) brainstormed a few ideas on gitter on how to do this...
> (1) Put more state/logic in the TaskInstance and Scheduler like this:
> my_task = PythonOperator(
>     task_id='my_task',
>     python_callable=my_command_that_access_the_datasource,
>     provide_context=True,
>     dag=dag,
>     blackout=my_blackout_logic_for_the_datasource # <---
>     )
> where my_blackout_logic is some function I provide that the scheduler calls to determine
whether or not it is the blackout period.
> (2) Pause DAGs on nightly basis. This can be done with the `pause_dag` CLI command scheduled
by cron / Jenkins. However could this be considered a core feature to bring into the Airflow
UI and scheduling system?

This message was sent by Atlassian JIRA

View raw message