airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ramandumcs@gmail.com <ramandu...@gmail.com>
Subject Scheduler gets slow with increasing number of Dags
Date Wed, 23 May 2018 16:22:21 GMT
Hi All,

We have a use case where there are 100s of DAGs in the scheduler's local dag folder but at
a time only ~50 dags are active(Other dags are disabled). New dags keep on adding to the local
Dag folder.
We are observing that scheduler is taking lot of time(around 20 minutes) in picking those
newly created dags.
We are using following conf
Machine config: 8 core, 16 GB RAM
Executor: LocalExecutor
Parallelism: 50
max_thread: 4,
Dir list interval: 60.

Is there any recommended setting or config to optimise the scheduler's dag listing and parsing.

Thanks,
Raman

Mime
View raw message