airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy Cooper <andy.coo...@astronomer.io>
Subject INFO REQUEST: Scheduler Multiprocessing
Date Tue, 05 Dec 2017 14:48:28 GMT
When running Airflow with a single dynamic DAG file generating multiple DAGs Airflow the scheduler
no longer forks processes on a per DAG basis and instead only forks on a per DAG file basis.
We are trying to understand why this decision was made and if there are any suggestions on
how to improve scheduler performance when an airflow instance contains one or more DAG files
that generate many DAGs.

-Andy Cooper
Mime
View raw message