airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (Jira)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-2167) Scheduler's clear_nonexistent_import_errors function should be called on first iteration
Date Wed, 11 Sep 2019 16:28:01 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-2167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927771#comment-16927771
] 

ASF GitHub Bot commented on AIRFLOW-2167:
-----------------------------------------

stale[bot] commented on pull request #3088: [AIRFLOW-2167] cleanup nonexistent import errors
on the first iteration in addition to periodically
URL: https://github.com/apache/airflow/pull/3088
 
 
   
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Scheduler's clear_nonexistent_import_errors function should be called on first iteration
> ----------------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-2167
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2167
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: scheduler
>    Affects Versions: 1.9.0
>            Reporter: Casey
>            Assignee: Casey
>            Priority: Minor
>         Attachments: Screen Shot 2018-03-02 at 2.08.29 PM.png
>
>
> In `airflow/jobs.py`, the `**clear_nonexistent_import_errors` function is not called
until the amount of seconds defined by `dag_dir_list_interval` has elapsed.  If the scheduler
is not alive for the duration of `dag_dir_list_interval` (300 seconds) this cleanup never
occurs.  In some environments this could result in error messages displaying on the UI permanently,
even if the DAG has been removed from the environment.
> It was previously an Airflow best practice to have the scheduler run N runtimes and terminate. 
Then, the scheduler would started again by an auxiliary process like Docker or Supervisor. 
This situation is what brought the bug to my attention.
> My suggested fix is to tweak jobs.py to run the import error cleanup on the first iteration
and periodically as defined by `dag_dir_list_interval`.  This way, a scheduler setup with
a small number of runs will still have old errors cleaned up.
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Mime
View raw message