airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <>
Subject [jira] [Commented] (AIRFLOW-910) Parallelize dag runs in backfills
Date Mon, 13 Mar 2017 02:50:05 GMT


ASF subversion and git services commented on AIRFLOW-910:

Commit dcc8ede5c1a2f6819b151dd5ce839f0a0917313a in incubator-airflow's branch refs/heads/v1-8-test
from [~bolke]
[;h=dcc8ede ]

[AIRFLOW-910] Use parallel task execution for backfills

The refactor to use dag runs in backfills caused a
in task execution performance as dag runs were
sequentially. Next to that, the backfills were non
due to the random execution of tasks, causing root
being added to the non ready list too soon.

This updates the backfill logic as follows:
* Parallelize execution of tasks
* Use a leave first execution model
* Replace state updates from the executor by task
based only

Closes #2107 from bolkedebruin/AIRFLOW-910

> Parallelize dag runs in backfills
> ---------------------------------
>                 Key: AIRFLOW-910
>                 URL:
>             Project: Apache Airflow
>          Issue Type: Sub-task
>          Components: backfill
>    Affects Versions: 1.8.0rc4
>            Reporter: Bolke de Bruin
>            Assignee: Bolke de Bruin
>            Priority: Blocker
>             Fix For: 1.8.0
> Currently dag runs are executed sequentially while backfilling. This is a regression
and slows down the processing off tasks.
> [~aoen]

This message was sent by Atlassian JIRA

View raw message