airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Casey (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AIRFLOW-2167) Scheduler's clear_nonexistent_import_errors function should be called on first iteration
Date Fri, 02 Mar 2018 19:18:00 GMT
Casey created AIRFLOW-2167:
------------------------------

             Summary: Scheduler's clear_nonexistent_import_errors function should be called
on first iteration
                 Key: AIRFLOW-2167
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2167
             Project: Apache Airflow
          Issue Type: Bug
          Components: scheduler
    Affects Versions: 1.9.0
            Reporter: Casey
            Assignee: Casey
         Attachments: Screen Shot 2018-03-02 at 2.08.29 PM.png

In `airflow/jobs.py`, the `**clear_nonexistent_import_errors` function is not called until
the amount of seconds defined by `dag_dir_list_interval` has elapsed.  If the scheduler is
not alive for the duration of `dag_dir_list_interval` (300 seconds) this cleanup never occurs. 
In some environments this could result in error messages displaying on the UI permanently,
even if the DAG has been removed from the environment.

It was previously an Airflow best practice to have the scheduler run N runtimes and terminate. 
Then, the scheduler would started again by an auxiliary process like Docker or Supervisor. 
This situation is what brought the bug to my attention.

My suggested fix is to tweak jobs.py to run the import error cleanup on the first iteration
and periodically as defined by `dag_dir_list_interval`.  This way, a scheduler setup with
a small number of runs will still have old errors cleaned up.

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message