airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Yang (JIRA)" <>
Subject [jira] [Commented] (AIRFLOW-435) Multiprocessing Scheduler is very slow
Date Wed, 17 Aug 2016 18:09:21 GMT


Paul Yang commented on AIRFLOW-435:

1636 introduced the parameter min_file_process_interval in

that controls how often DAGs are processed / scheduled. The default is 3 minutes - if you
want a smaller delay, you can set this to a smaller value. Also, what's your max_threads set
to? I'll file a follow up PR to perhaps tweak this value and add missing documentation.

> Multiprocessing Scheduler is very slow
> --------------------------------------
>                 Key: AIRFLOW-435
>                 URL:
>             Project: Apache Airflow
>          Issue Type: Bug
>            Reporter: George Leslie-Waksman
>            Assignee: Paul Yang
> The PR has dramatically slowed
down the scheduler. Running code prior to 1636 will result in rapid scheduling of many tasks.
After 1636, tasks can wait in a null state for minutes without being scheduled.

This message was sent by Atlassian JIRA

View raw message