airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dan Davydov (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AIRFLOW-536) Dags only get one dagrun is scheduled per scheduler loop
Date Fri, 30 Sep 2016 19:57:20 GMT
Dan Davydov created AIRFLOW-536:
-----------------------------------

             Summary: Dags only get one dagrun is scheduled per scheduler loop
                 Key: AIRFLOW-536
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-536
             Project: Apache Airflow
          Issue Type: Bug
            Reporter: Dan Davydov


At the moment Dags only get one dag scheduled per scheduler loop (which can be quite slow,
on the order of minutes), but what should happen is that all pending dagruns for a dagrun
should be scheduled.

In addition to dagruns being scheduled more quickly for dags that havem ultiple dag runs queued
up (e.g. paused dags), fixing this issue will enable a "only_run_latest" option for dags (so
e.g. if the dag is paused and then later unpaused, only the most recent ds will be run), as
well as facilitating compatibility between backfills and the scheduler (if you run a backfill
over 100 dag runs the backfill will be waiting on 100 scheduler cycles).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message