airflow-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reed Villanueva <rvillanu...@ucera.org>
Subject Re: Use separate environ and sys.path between dags
Date Thu, 17 Oct 2019 22:37:19 GMT
Interesting, thanks for the info.

On Thu, Oct 17, 2019 at 12:11 PM Kamil BreguĊ‚a <kamil.bregula@polidea.com>
wrote:

> Airflow does not have a built-in mechanism, but if you have the
> opportunity to create a contribution then you should look in the
> direction of BaseTaskRunner.
>
> On Thu, Oct 17, 2019 at 9:57 PM Reed Villanueva <rvillanueva@ucera.org>
> wrote:
> >
> > Is there a way to have dags use separate environ and sys.path values?
> >
> > For example, I have files dag1.py and dag2.py that both want to import
> modules and run files as tasks from directories /files/for/dag1 and
> /files/for/dag2. The files are named the same in both dirs but do very
> different things. Currently, if I set environ (eg. PROJECT_HOME) and
> sys.path in one dag file, I see that it affects all other dag files in
> ~/airflow/dag. This causes problems / mixups when trying to import modules
> or reference files in dirs where files have same names.
> >
> > Is there a way to get this working without using packaged dags? Can't
> use packaged dags, since have some bash scripts that run as tasks that use
> environ variables (like PROJECT_HOME) to reference local dirs for other
> things and as I understand it a packaged dag will only help co-locating the
> files, but not isolate env vars.
> >
> > This electronic message is intended only for the named
> > recipient, and may contain information that is confidential or
> > privileged. If you are not the intended recipient, you are
> > hereby notified that any disclosure, copying, distribution or
> > use of the contents of this message is strictly prohibited. If
> > you have received this message in error or are not the named
> > recipient, please notify us immediately by contacting the
> > sender at the electronic mail address noted above, and delete
> > and destroy all copies of this message. Thank you.
>

-- 
This electronic message is intended only for the named 
recipient, and may 
contain information that is confidential or 
privileged. If you are not the 
intended recipient, you are 
hereby notified that any disclosure, copying, 
distribution or 
use of the contents of this message is strictly 
prohibited. If 
you have received this message in error or are not the 
named
recipient, please notify us immediately by contacting the 
sender at 
the electronic mail address noted above, and delete 
and destroy all copies 
of this message. Thank you.

Mime
View raw message