airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-2203) Dag import too slow for very large dags
Date Wed, 14 Mar 2018 08:16:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398223#comment-16398223
] 

ASF subversion and git services commented on AIRFLOW-2203:
----------------------------------------------------------

Commit 81ec595b6c1ac05bc7f42e2c92c0dd79409953a4 in incubator-airflow's branch refs/heads/master
from [~wongwill]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=81ec595 ]

[AIRFLOW-2203] Cache signature in apply_defaults

Cache inspect.signature for the wrapper closure to avoid calling it at
every decorated invocation. This is separate sig_cache created per
decoration, i.e. each function decorated using apply_defaults will have
a different sig_cache.


> Dag import too slow for very large dags
> ---------------------------------------
>
>                 Key: AIRFLOW-2203
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2203
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: DAG
>    Affects Versions: 1.8.1, 1.8.0, 1.9.0, 1.9.1
>            Reporter: Will Wong
>            Assignee: Will Wong
>            Priority: Major
>              Labels: performance, usability
>
> Dag import for large dags is too slow. It is very easy to timeout very large dags with
< 500 tasks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message