airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Badger (JIRA)" <>
Subject [jira] [Updated] (AIRFLOW-2393) UI tree view struggles with large dags (60 tasks)x25 dag histories
Date Mon, 30 Apr 2018 14:10:00 GMT


Badger updated AIRFLOW-2393:
    Affects Version/s: 1.9.0

> UI tree view struggles with large dags (60 tasks)x25 dag histories
> ------------------------------------------------------------------
>                 Key: AIRFLOW-2393
>                 URL:
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: ui
>    Affects Versions: 1.9.0
>            Reporter: Badger
>            Priority: Major
> Hi, 
> We are noticing the tree view is taking a long time to render as our DAG has become more
complex. We will need to start breaking our dag apart in order to continue to use the user
> The basic problem is that a reasonably complex DAG (60 operators) x the standard 25 dag
run histories on the tree view causes a 350MB json response (compressed to 8MB) to be downloaded,
this then needs the browser to render it.
> On quick observation this appears to be because, the response appears to contain all
meta-data for each task.
> Is this something others think is a problem. We occasionally have to refresh due to memory
errors and have already increased the RAM allocated to the box.
> A suggestion might be to load specific instance history when a user hovers over the task,
rather than exporting all of the history on page load. I'd look at contributing a PR but
haven't had chance to take a look at this area of the code base.
> Thanks

This message was sent by Atlassian JIRA

View raw message