airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Yang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-435) Multiprocessing Scheduler is very slow
Date Wed, 17 Aug 2016 18:09:21 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425078#comment-15425078
] 

Paul Yang commented on AIRFLOW-435:
-----------------------------------

1636 introduced the parameter min_file_process_interval in 

https://github.com/apache/incubator-airflow/blob/master/airflow/configuration.py#L320

that controls how often DAGs are processed / scheduled. The default is 3 minutes - if you
want a smaller delay, you can set this to a smaller value. Also, what's your max_threads set
to? I'll file a follow up PR to perhaps tweak this value and add missing documentation.

> Multiprocessing Scheduler is very slow
> --------------------------------------
>
>                 Key: AIRFLOW-435
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-435
>             Project: Apache Airflow
>          Issue Type: Bug
>            Reporter: George Leslie-Waksman
>            Assignee: Paul Yang
>
> The PR https://github.com/apache/incubator-airflow/pull/1636 has dramatically slowed
down the scheduler. Running code prior to 1636 will result in rapid scheduling of many tasks.
After 1636, tasks can wait in a null state for minutes without being scheduled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message