Mailing-List: contact commits-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
Date: Mon, 9 May 2016 18:58:12 +0000 (UTC)
From: "Chris Riccomini (JIRA)" <jira@apache.org>
To: commits@airflow.incubator.apache.org
Message-ID: <JIRA.12966259.1462819960000.145114.1462820292872@Atlassian.JIRA>
In-Reply-To: <JIRA.12966259.1462819960000@Atlassian.JIRA>
References: <JIRA.12966259.1462819960000@Atlassian.JIRA> <JIRA.12966259.1462819960068@arcas>
Subject: [jira] [Commented] (AIRFLOW-81) Scheduler blackout time period
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 09 May 2016 18:58:16 -0000


    [ https://issues.apache.org/jira/browse/AIRFLOW-81?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15276817#comment-15276817 ] 

Chris Riccomini commented on AIRFLOW-81:
----------------------------------------

What do you think about writing a new sensor based off of the TimeSensor that blocks during the blackout period?

> Scheduler blackout time period
> ------------------------------
>
>                 Key: AIRFLOW-81
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-81
>             Project: Apache Airflow
>          Issue Type: Wish
>          Components: scheduler
>            Reporter: Sean McIntyre
>            Priority: Minor
>              Labels: features
>
> I have the need for a scheduler blackout time period in Airflow.
> My team, which uses Airflow, has been asked to not query one of my company's data sources between midnight and 7 AM. When we launch big backfills on this data source, it would be nice to have the Scheduler not schedule some TaskInstances during the blackout hours.
> We (@r39132 and @ledsusop) brainstormed a few ideas on gitter on how to do this...
> (1) Put more state/logic in the TaskInstance and Scheduler like this:
> my_task = PythonOperator(
>     task_id='my_task',
>     python_callable=my_command_that_access_the_datasource,
>     provide_context=True,
>     dag=dag,
>     blackout=my_blackout_logic_for_the_datasource # <---
>     )
> where my_blackout_logic is some function I provide that the scheduler calls to determine whether or not it is the blackout period.
> (2) Pause DAGs on nightly basis. This can be done with the `pause_dag` CLI command scheduled by cron / Jenkins. However could this be considered a core feature to bring into the Airflow UI and scheduling system?


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)