Mailing-List: contact commits-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
Date: Mon, 13 Mar 2017 02:50:05 +0000 (UTC)
From: "ASF subversion and git services (JIRA)" <jira@apache.org>
To: commits@airflow.incubator.apache.org
Message-ID: <JIRA.13046302.1488047085000.13305.1489373405077@Atlassian.JIRA>
In-Reply-To: <JIRA.13046302.1488047085000@Atlassian.JIRA>
References: <JIRA.13046302.1488047085000@Atlassian.JIRA> <JIRA.13046302.1488047085146@jira-lw-us.apache.org>
Subject: [jira] [Commented] (AIRFLOW-910) Parallelize dag runs in backfills
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 13 Mar 2017 02:50:09 -0000


    [ https://issues.apache.org/jira/browse/AIRFLOW-910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15906795#comment-15906795 ] 

ASF subversion and git services commented on AIRFLOW-910:
---------------------------------------------------------

Commit dcc8ede5c1a2f6819b151dd5ce839f0a0917313a in incubator-airflow's branch refs/heads/v1-8-test from [~bolke]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=dcc8ede ]

[AIRFLOW-910] Use parallel task execution for backfills

The refactor to use dag runs in backfills caused a
regression
in task execution performance as dag runs were
executed
sequentially. Next to that, the backfills were non
deterministic
due to the random execution of tasks, causing root
tasks
being added to the non ready list too soon.

This updates the backfill logic as follows:
* Parallelize execution of tasks
* Use a leave first execution model
* Replace state updates from the executor by task
based only

Closes #2107 from bolkedebruin/AIRFLOW-910


> Parallelize dag runs in backfills
> ---------------------------------
>
>                 Key: AIRFLOW-910
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-910
>             Project: Apache Airflow
>          Issue Type: Sub-task
>          Components: backfill
>    Affects Versions: 1.8.0rc4
>            Reporter: Bolke de Bruin
>            Assignee: Bolke de Bruin
>            Priority: Blocker
>             Fix For: 1.8.0
>
>
> Currently dag runs are executed sequentially while backfilling. This is a regression and slows down the processing off tasks.
> [~aoen]


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)