airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <>
Subject [jira] [Commented] (AIRFLOW-69) Make backfills use DagRuns
Date Fri, 12 Aug 2016 12:49:20 GMT


ASF subversion and git services commented on AIRFLOW-69:

Commit c39e4f6aa45d0f7581410e9c0f42ab56cbf30785 in incubator-airflow's branch refs/heads/master
from [~bolke]
[;h=c39e4f6 ]

[AIRFLOW-69] Use dag runs in backfill jobs

Backfill jobs create taskinstances without any associated
DagRuns. This creates consistency errors. This patch addresses
this issue and also makes the scheduler backfill aware.

The scheduler makes sure to schedule new dag runs after the
last dag run including backfills. It will not pick up
any tasks that are part of a backfill as those are considered
to be managed by the backfill process. This can be (and should
be) changed when backfill are running in a scheduled fashion.

It doesn't deal with the remaining issue that backfills can be
scheduled on top of existing dag runs and that due to this
TaskInstances can point to multiple DagRuns.

Closes #1667 from bolkedebruin/backfill_dagrun

> Make backfills use DagRuns
> --------------------------
>                 Key: AIRFLOW-69
>                 URL:
>             Project: Apache Airflow
>          Issue Type: Sub-task
>          Components: scheduler
>    Affects Versions: Airflow 1.7.1
>            Reporter: Bolke de Bruin
>             Fix For: Airflow 1.8

This message was sent by Atlassian JIRA

View raw message