airflow-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reed Villanueva <>
Subject Use separate environ and sys.path between dags
Date Thu, 17 Oct 2019 19:54:45 GMT
Is there a way to have dags use separate environ and sys.path values?

For example, I have files and that both want to import
modules and run files as tasks from directories /files/for/dag1 and
/files/for/dag2. The files are named the same in both dirs but do very
different things. Currently, if I set environ (eg. PROJECT_HOME) and
sys.path in one dag file, I see that it affects all other dag files in
~/airflow/dag. This causes problems / mixups when trying to import modules
or reference files in dirs where files have same names.

Is there a way to get this working without using packaged dags? Can't use
packaged dags, since have some bash scripts that run as tasks that use
environ variables (like PROJECT_HOME) to reference local dirs for other
things and as I understand it a packaged dag will only help co-locating the
files, but not isolate env vars.

This electronic message is intended only for the named 
recipient, and may 
contain information that is confidential or 
privileged. If you are not the 
intended recipient, you are 
hereby notified that any disclosure, copying, 
distribution or 
use of the contents of this message is strictly 
prohibited. If 
you have received this message in error or are not the 
recipient, please notify us immediately by contacting the 
sender at 
the electronic mail address noted above, and delete 
and destroy all copies 
of this message. Thank you.

View raw message