airflow-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kamil BreguĊ‚a <kamil.breg...@polidea.com>
Subject Re: Use separate environ and sys.path between dags
Date Thu, 17 Oct 2019 22:11:24 GMT
Airflow does not have a built-in mechanism, but if you have the
opportunity to create a contribution then you should look in the
direction of BaseTaskRunner.

On Thu, Oct 17, 2019 at 9:57 PM Reed Villanueva <rvillanueva@ucera.org> wrote:
>
> Is there a way to have dags use separate environ and sys.path values?
>
> For example, I have files dag1.py and dag2.py that both want to import modules and run
files as tasks from directories /files/for/dag1 and /files/for/dag2. The files are named the
same in both dirs but do very different things. Currently, if I set environ (eg. PROJECT_HOME)
and sys.path in one dag file, I see that it affects all other dag files in ~/airflow/dag.
This causes problems / mixups when trying to import modules or reference files in dirs where
files have same names.
>
> Is there a way to get this working without using packaged dags? Can't use packaged dags,
since have some bash scripts that run as tasks that use environ variables (like PROJECT_HOME)
to reference local dirs for other things and as I understand it a packaged dag will only help
co-locating the files, but not isolate env vars.
>
> This electronic message is intended only for the named
> recipient, and may contain information that is confidential or
> privileged. If you are not the intended recipient, you are
> hereby notified that any disclosure, copying, distribution or
> use of the contents of this message is strictly prohibited. If
> you have received this message in error or are not the named
> recipient, please notify us immediately by contacting the
> sender at the electronic mail address noted above, and delete
> and destroy all copies of this message. Thank you.

Mime
View raw message