airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dan Davydov (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (AIRFLOW-1311) Improve Webserver Load Time For Large DAGs
Date Wed, 16 Aug 2017 00:19:00 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-1311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Dan Davydov reassigned AIRFLOW-1311:
------------------------------------

    Assignee: Edgar Rodriguez  (was: Dan Davydov)

> Improve Webserver Load Time For Large DAGs
> ------------------------------------------
>
>                 Key: AIRFLOW-1311
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1311
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: webserver
>            Reporter: Dan Davydov
>            Assignee: Edgar Rodriguez
>
> Large DAGs can take an extremely long time to load in the Airflow UI (minutes/timeout).
> The fixes are as follows:
> 1. Lazy load DAGs (load up to a certain # of tasks by default, prioritizing tasks by
their depth, and allow users to expand sections for these DAGs, ideally prefetch deeper tasks
once the initial set of tasks has rendered )
> 2. Identify bottlenecks/performance issues in both the frontend/backend for rendering
DAGs on the webserver and fix them. Airflow should be more performant for displaying DAGs
that are somewhat large, e.g. DAGs that have up to 500 nodes and 2000 edges (dependencies
from one task to another) should render within a couple of seconds.
> 3. Make DAG loading asynchronous in the UI (once the top-level tasks have loaded display
them immediately). We might not want to do this as users might try to click something only
to have the UI change from underneath them
> [~saguziel]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message