airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dan Davydov (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-2027) Only trigger sleep in scheduler after all files have parsed
Date Mon, 29 Jan 2018 20:04:00 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-2027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Dan Davydov updated AIRFLOW-2027:
---------------------------------
    Description: 
The scheduler loop sleeps for 1 second every loop unnecessarily. Remove this sleep to slightly
speed up scheduling, and instead do it once all files have been parsed. It can add up since
it runs to every scheduler loop which runs # of dags to parse/scheduler parallelism times.

Also remove the unnecessary increased file processing interval in tests which slows them down.

  was:
The scheduler loop sleeps for 1 second every loop unnecessarily. Remove this sleep to slightly
speed up scheduling. It can add up since it runs to every scheduler loop which runs # of
dags to parse/scheduler parallelism times.

 

Also remove the unnecessary increased file processing interval in tests which slows them down.


> Only trigger sleep in scheduler after all files have parsed
> -----------------------------------------------------------
>
>                 Key: AIRFLOW-2027
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2027
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: scheduler
>            Reporter: Dan Davydov
>            Assignee: Dan Davydov
>            Priority: Major
>
> The scheduler loop sleeps for 1 second every loop unnecessarily. Remove this sleep to
slightly speed up scheduling, and instead do it once all files have been parsed. It can add
up since it runs to every scheduler loop which runs # of dags to parse/scheduler parallelism
times.
> Also remove the unnecessary increased file processing interval in tests which slows them
down.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message