airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boris Tyukin <bo...@boristyukin.com>
Subject variable scope with dynamic dags
Date Wed, 22 Mar 2017 12:43:57 GMT
Hi,

I have a weird question but it bugs my mind. I have some like below to
generate dags dynamically, using Max's example code from FAQ.

It works fine but I have one large dict (let's call it my_outer_dict) that
takes over 60Mb in memory and I need to access it from all generated dags.
Needless to say, i do not want to recreate that dict for every dag as I
want to load it to memory only once.

To my surprise, if i define that dag outside of my dag definition code, I
can still access it.

Can someone explain why and where is it stored? I thought only dag
definitions are loaded to dagbag and not the variables outside it.

Is it even a good practice and will it work still if I switch to celery
executor?


def get_dag(i):
    dag_id = 'foo_{}'.format(i)
dag = DAG(dag_id)
....
print my_outer_dict

my_outer_dict = {}
for i in range(10):
dag = get_dag(i)
    globals()[dag.dag_id] = dag

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message