airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dan Davydov (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-536) Dags only get one dagrun is scheduled per scheduler loop
Date Tue, 29 Nov 2016 20:21:58 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Dan Davydov updated AIRFLOW-536:
--------------------------------
    Description: 
At the moment Dags only get one dag scheduled per scheduler loop (which can be quite slow,
on the order of minutes), but what should happen is that all pending dagruns for a dag should
be scheduled.

In addition to dagruns being scheduled more quickly for dags that have multiple dag runs queued
up (e.g. previously paused dags), fixing this issue will enable a "only_run_latest" option
for dags (so e.g. if the dag is paused and then later unpaused, only the most recent ds will
be run), as well as facilitating compatibility between backfills and the scheduler (if you
run a backfill over 100 dag runs the backfill will be waiting on 100 scheduler cycles).

  was:
At the moment Dags only get one dag scheduled per scheduler loop (which can be quite slow,
on the order of minutes), but what should happen is that all pending dagruns for a dag should
be scheduled.

In addition to dagruns being scheduled more quickly for dags that have multiple dag runs queued
up (e.g. paused dags), fixing this issue will enable a "only_run_latest" option for dags (so
e.g. if the dag is paused and then later unpaused, only the most recent ds will be run), as
well as facilitating compatibility between backfills and the scheduler (if you run a backfill
over 100 dag runs the backfill will be waiting on 100 scheduler cycles).


> Dags only get one dagrun is scheduled per scheduler loop
> --------------------------------------------------------
>
>                 Key: AIRFLOW-536
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-536
>             Project: Apache Airflow
>          Issue Type: Bug
>            Reporter: Dan Davydov
>
> At the moment Dags only get one dag scheduled per scheduler loop (which can be quite
slow, on the order of minutes), but what should happen is that all pending dagruns for a dag
should be scheduled.
> In addition to dagruns being scheduled more quickly for dags that have multiple dag runs
queued up (e.g. previously paused dags), fixing this issue will enable a "only_run_latest"
option for dags (so e.g. if the dag is paused and then later unpaused, only the most recent
ds will be run), as well as facilitating compatibility between backfills and the scheduler
(if you run a backfill over 100 dag runs the backfill will be waiting on 100 scheduler cycles).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message