airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-2614) Airflow trigger_run API is very slow
Date Wed, 11 Jul 2018 09:59:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-2614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16539815#comment-16539815
] 

ASF subversion and git services commented on AIRFLOW-2614:
----------------------------------------------------------

Commit aa17b5baf886f7e23b3a9bcf8fef45759193f1b6 in incubator-airflow's branch refs/heads/master
from [~mishikaps]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=aa17b5b ]

[AIRFLOW-2614] Speed up trigger_dag API call when lots of DAGs in system

Rather than loading all dags in the DagBag, find the path to the
specific DAG from the ORM and load only that one.

Closes #3590 from mishikaSingh/master


> Airflow trigger_run API is very slow
> ------------------------------------
>
>                 Key: AIRFLOW-2614
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2614
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: DagRun
>    Affects Versions: Airflow 2.0, 1.9.0
>            Reporter: raman
>            Assignee: Mishika Singh
>            Priority: Major
>             Fix For: 2.0.0
>
>
> Current implementation of trigger_dag processes all Local dag files sequentially before
creating a DAG run. Its done inside trigger_dag function in trigger_dag.py
> "def trigger_dag(dag_id, run_id=None, conf=None, execution_date=None):
>  dagbag = DagBag()
> ....."
> Processing all the files to get the dagBag slows down the trigger_dag api and increases
latency. We have observed that it starts taking 10(s) of seconds as number of Dag Files increase. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message