airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-2203) Dag import too slow for very large dags
Date Wed, 14 Mar 2018 08:16:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398225#comment-16398225
] 

ASF subversion and git services commented on AIRFLOW-2203:
----------------------------------------------------------

Commit c3730650c852cd7a5e06a5933f5064bbb04e0e88 in incubator-airflow's branch refs/heads/master
from [~wongwill]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=c373065 ]

[AIRFLOW-2203] Defer cycle detection

Moved from adding_task to when dag is being bagged.
This changes import dag runtime from polynomial to somewhat linear.

Closes #3116 from wongwill86:dag_import_speed


> Dag import too slow for very large dags
> ---------------------------------------
>
>                 Key: AIRFLOW-2203
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2203
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: DAG
>    Affects Versions: 1.8.1, 1.8.0, 1.9.0, 1.9.1
>            Reporter: Will Wong
>            Assignee: Will Wong
>            Priority: Major
>              Labels: performance, usability
>
> Dag import for large dags is too slow. It is very easy to timeout very large dags with
< 500 tasks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message