airflow-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reed Villanueva <rvillanu...@ucera.org>
Subject How to manage python packages between airflow dags?
Date Wed, 16 Oct 2019 00:03:02 GMT
If I have multiple airflow dags with some overlapping python package
dependencies, how can I keep each of these project deps. decoupled? Eg. if
I had project A and B on same server I would run each of them with
something like...

source /path/to/virtualenv_a/activate
python script_a.py
deactivate
source /path/to/virtualenv_b/activate
python script_b.py
deactivate

Basically, would like to run dags with the same situation (eg. each dag
uses python scripts that have may have overlapping package deps. that I
would like to develop separately (ie. not have to update all code using a
package when want to update the package just for one project)). Note, I've
been using the BashOperator to run python tasks like...

do_stuff = BashOperator(
        task_id='my_task',
        bash_command='python /path/to/script.py'),
        execution_timeout=timedelta(minutes=30),
        dag=dag)

Is there a way to get this working? Is there some other best-practice way
that airflow intendeds for people to address (or avoid) these kinds of
problems?

-- 
This electronic message is intended only for the named 
recipient, and may 
contain information that is confidential or 
privileged. If you are not the 
intended recipient, you are 
hereby notified that any disclosure, copying, 
distribution or 
use of the contents of this message is strictly 
prohibited. If 
you have received this message in error or are not the 
named
recipient, please notify us immediately by contacting the 
sender at 
the electronic mail address noted above, and delete 
and destroy all copies 
of this message. Thank you.

Mime
View raw message