airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kaxil Naik (Jira)" <>
Subject [jira] [Updated] (AIRFLOW-4924) Loading DAGs asynchronously in Airflow webserver
Date Thu, 29 Aug 2019 22:27:00 GMT


Kaxil Naik updated AIRFLOW-4924:
    Fix Version/s:     (was: 1.10.5)

> Loading DAGs asynchronously in Airflow webserver
> ------------------------------------------------
>                 Key: AIRFLOW-4924
>                 URL:
>             Project: Apache Airflow
>          Issue Type: New Feature
>          Components: webserver
>    Affects Versions: 1.10.4
>            Reporter: Zhou Fang
>            Assignee: Zhou Fang
>            Priority: Major
>              Labels: features, scalability, webserver
>             Fix For: 1.10.6
>   Original Estimate: 336h
>  Remaining Estimate: 336h
> h2. Scalability Issue in Webserver
> Airflow webserver uses gunicorn workers to serve HTTP requests. It loads all DAGs from
DAG files before serving requests. If there are many DAGs (e.g., > 1,000), loading all
DAGs can take a significant amount of time.
> Airflow webserver also relies on restarting gunicorn workers to refresh all DAGs. This
refreshing interval is set by webserver-worker_refresh_interval, default to 30s. As a result,
if loading all DAGs takes >30s, the webserver will never be ready for HTTP requests.
> The current solution is to skip loading DAGs by using env var SKIP_DAGS_PARSING. It
makes the webserver work, but there is no DAG on the UI.
> h2. Asynchronously DAG Loading
> The solution here is to load DAGs asynchronously in the background. It creates a background
process to load DAGs, stringifies DAGs, and sends DAGs to gunicorn worker process. The stringifying
step is needed because some fields can not be pickled, e.g., locally defined functions and
user defined modules. It aggressively transform all fields of DAG and task to be string-compatible.
> This feature is enabled by webserver-async_dagbag_loader=True. The background process
sends DAGs to gunicorn worker gradually (every webserver-dagbag_sync_interval). DAG refreshing
interval is controlled by webserver-collect_dags_interval.
> Asynchronous DAG loading has been released in Google Cloud Composer as an Alpha feature:
>  []
>  []
> This issue is created to merge the feature to Airflow upstream.

This message was sent by Atlassian Jira

View raw message