airflow-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reed Villanueva <rvillanu...@ucera.org>
Subject When does a airflow dag definition get evaluated?
Date Mon, 28 Oct 2019 19:44:06 GMT
Suppose I have an airflow dag file that creates a graph like so...

def get_current_info(filename)
    current_info = {}
    <fill in info in current_info relevant for today's date for given file>
    return current_info

files = [
    get_current_info("file_001"),
    get_current_info("file_002"),
    ....
]

for f in files:
    <some BashOperator bo1 using f's current info dict>
    <some BashOperator bo2 using f's current info dict>
    ....

    bo1 >> bo2
    ....

Since these values in the current_info dict that is used to define the dag
changes periodically (here, daily), I would like to know by what process /
schedule the dag definition gets updated. (I print the current_info values
each run and values appear to be updating, but curious as to how and when
exactly this happens).

When does an airflow dag definition get evaluated? referenced anywhere in
the docs?

-- 
This electronic message is intended only for the named 
recipient, and may 
contain information that is confidential or 
privileged. If you are not the 
intended recipient, you are 
hereby notified that any disclosure, copying, 
distribution or 
use of the contents of this message is strictly 
prohibited. If 
you have received this message in error or are not the 
named
recipient, please notify us immediately by contacting the 
sender at 
the electronic mail address noted above, and delete 
and destroy all copies 
of this message. Thank you.

Mime
View raw message