airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mishika Singh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-2614) Airflow trigger_run API is very slow
Date Tue, 10 Jul 2018 12:12:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-2614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16538486#comment-16538486
] 

Mishika Singh commented on AIRFLOW-2614:
----------------------------------------

raised PR for this : [https://github.com/apache/incubator-airflow/pull/3590]

 

> Airflow trigger_run API is very slow
> ------------------------------------
>
>                 Key: AIRFLOW-2614
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2614
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: DagRun
>    Affects Versions: Airflow 2.0, 1.9.0
>            Reporter: raman
>            Priority: Major
>
> Current implementation of trigger_dag processes all Local dag files sequentially before
creating a DAG run. Its done inside trigger_dag function in trigger_dag.py
> "def trigger_dag(dag_id, run_id=None, conf=None, execution_date=None):
>  dagbag = DagBag()
> ....."
> Processing all the files to get the dagBag slows down the trigger_dag api and increases
latency. We have observed that it starts taking 10(s) of seconds as number of Dag Files increase. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message