airflow-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kamil BreguĊ‚a <kamil.breg...@polidea.com>
Subject Re: How to manage python packages between airflow dags?
Date Wed, 16 Oct 2019 09:19:07 GMT
Don't use a wrapper, because only what it does changes the value of
the PATH variable.
11:16 $ type python
python is /Library/Frameworks/Python.framework/Versions/2.7/bin/python
11:16 $ workon airflow
11:16 $ type python
python is /Users/kamilbregula/.virtualenvs/airflow/bin/python

so if you want to use a different environment then you can run the
script using the interpreter's direct specification.
For example:
/Users/kamilbregula/.virtualenvs/airflow/bin/python /path/to/script.py

On Wed, Oct 16, 2019 at 2:06 AM Reed Villanueva <rvillanueva@ucera.org> wrote:
>
> If I have multiple airflow dags with some overlapping python package dependencies, how
can I keep each of these project deps. decoupled? Eg. if I had project A and B on same server
I would run each of them with something like...
>
> source /path/to/virtualenv_a/activate
> python script_a.py
> deactivate
> source /path/to/virtualenv_b/activate
> python script_b.py
> deactivate
>
> Basically, would like to run dags with the same situation (eg. each dag uses python scripts
that have may have overlapping package deps. that I would like to develop separately (ie.
not have to update all code using a package when want to update the package just for one project)).
Note, I've been using the BashOperator to run python tasks like...
>
> do_stuff = BashOperator(
>         task_id='my_task',
>         bash_command='python /path/to/script.py'),
>         execution_timeout=timedelta(minutes=30),
>         dag=dag)
>
> Is there a way to get this working? Is there some other best-practice way that airflow
intendeds for people to address (or avoid) these kinds of problems?
>
>
> This electronic message is intended only for the named
> recipient, and may contain information that is confidential or
> privileged. If you are not the intended recipient, you are
> hereby notified that any disclosure, copying, distribution or
> use of the contents of this message is strictly prohibited. If
> you have received this message in error or are not the named
> recipient, please notify us immediately by contacting the
> sender at the electronic mail address noted above, and delete
> and destroy all copies of this message. Thank you.

Mime
View raw message