airflow-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reed Villanueva <rvillanu...@ucera.org>
Subject Re: How to manage python packages between airflow dags?
Date Wed, 16 Oct 2019 20:05:37 GMT
Good idea, thanks.
Works well for the modular way I am developing python scripts and modules.

On Tue, Oct 15, 2019 at 11:19 PM Kamil BreguĊ‚a <kamil.bregula@polidea.com>
wrote:

> Don't use a wrapper, because only what it does changes the value of
> the PATH variable.
> 11:16 $ type python
> python is /Library/Frameworks/Python.framework/Versions/2.7/bin/python
> 11:16 $ workon airflow
> 11:16 $ type python
> python is /Users/kamilbregula/.virtualenvs/airflow/bin/python
>
> so if you want to use a different environment then you can run the
> script using the interpreter's direct specification.
> For example:
> /Users/kamilbregula/.virtualenvs/airflow/bin/python /path/to/script.py
>
> On Wed, Oct 16, 2019 at 2:06 AM Reed Villanueva <rvillanueva@ucera.org>
> wrote:
> >
> > If I have multiple airflow dags with some overlapping python package
> dependencies, how can I keep each of these project deps. decoupled? Eg. if
> I had project A and B on same server I would run each of them with
> something like...
> >
> > source /path/to/virtualenv_a/activate
> > python script_a.py
> > deactivate
> > source /path/to/virtualenv_b/activate
> > python script_b.py
> > deactivate
> >
> > Basically, would like to run dags with the same situation (eg. each dag
> uses python scripts that have may have overlapping package deps. that I
> would like to develop separately (ie. not have to update all code using a
> package when want to update the package just for one project)). Note, I've
> been using the BashOperator to run python tasks like...
> >
> > do_stuff = BashOperator(
> >         task_id='my_task',
> >         bash_command='python /path/to/script.py'),
> >         execution_timeout=timedelta(minutes=30),
> >         dag=dag)
> >
> > Is there a way to get this working? Is there some other best-practice
> way that airflow intendeds for people to address (or avoid) these kinds of
> problems?
> >
> >
> > This electronic message is intended only for the named
> > recipient, and may contain information that is confidential or
> > privileged. If you are not the intended recipient, you are
> > hereby notified that any disclosure, copying, distribution or
> > use of the contents of this message is strictly prohibited. If
> > you have received this message in error or are not the named
> > recipient, please notify us immediately by contacting the
> > sender at the electronic mail address noted above, and delete
> > and destroy all copies of this message. Thank you.
>

-- 
This electronic message is intended only for the named 
recipient, and may 
contain information that is confidential or 
privileged. If you are not the 
intended recipient, you are 
hereby notified that any disclosure, copying, 
distribution or 
use of the contents of this message is strictly 
prohibited. If 
you have received this message in error or are not the 
named
recipient, please notify us immediately by contacting the 
sender at 
the electronic mail address noted above, and delete 
and destroy all copies 
of this message. Thank you.

Mime
View raw message