airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dan Davydov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-2027) Remove unnecessary 1s sleep in scheduler loop
Date Sat, 27 Jan 2018 00:49:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-2027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16341861#comment-16341861
] 

Dan Davydov commented on AIRFLOW-2027:
--------------------------------------

the PR I'm about to cut will sleep 1s - time it took to parse dags (so still 1s minimum).
Not greatly concerned about the busy looping really, but the log spam is an issue.

 

We have found in high load situations, 10% of the delay was caused by this sleep.

> Remove unnecessary 1s sleep in scheduler loop
> ---------------------------------------------
>
>                 Key: AIRFLOW-2027
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2027
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: scheduler
>            Reporter: Dan Davydov
>            Assignee: Dan Davydov
>            Priority: Major
>
> The scheduler loop sleeps for 1 second every loop unnecessarily. Remove this sleep to
slightly speed up scheduling. It can add up since it runs to every scheduler loop which runs
# of dags to parse/scheduler parallelism times.
>  
> Also remove the unnecessary increased file processing interval in tests which slows them
down.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message