airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joy Gao <j...@wepay.com>
Subject Re: Environment variables - Oracle LD_LIBRARY_PATH
Date Sat, 09 Dec 2017 01:07:13 GMT
Hi Rajesh,

Odd that it still doesn't work after setting the worker's env profile. We
run LocalExecutor so this has never been an issue. I will leave to others
to comment.

(One thing worth trying is to explicitly export the variable  `export
VAR=VAL` in a subprocess in the operator/hook so that it lives beyond the
scope of the python process.)

Cheers,
Joy


On Fri, Dec 8, 2017 at 4:32 PM, Rajesh Chamarthi <rajesh.chamarthi@gmail.com
> wrote:

> Hi Joy,
>
> Thank you, “airflow test” does not seem like a good way to test
> environment in that case :)
>
> Yes, I am running using celery executor and running the tasks in a
> distributed manner. I currently set the environment variable in the hook
> just before it tries to get a DB connection.
>
> We also use supervisord to manage the worker, scheduler and web server
> processes and I have changed the worker’s env profile to set the
> environment variable, but it does not work either.
>
>
> > On Dec 8, 2017, at 2:50 PM, Joy Gao <joyg@wepay.com> wrote:
> >
> > Hi Rajesh,
> >
> > `airflow test` executes the task synchronously in the same python process
> > (without touching the scheduler), which is probably why it went down the
> > happy path.
> >
> > Are you using celery executor and running the tasks in a distributed
> > manner? It* sounds like* the task execution env doesn't know about the
> > variable.
> >
> > Also depending on where you added the code to your custom hook, it may be
> > in a different process from which the task is executed, since os.environ
> > sets variable only for the scope/duration of the python process (and its
> > children).
> >
> >
> > On Fri, Dec 8, 2017 at 7:55 AM, Rajesh Chamarthi <
> rajesh.chamarthi@gmail.com
> >> wrote:
> >
> >> Hi all,
> >>
> >> What is the right way to set environment variables to be used in DAGS.
> >> Also, is it possible to export them such that currently running
> scheduler
> >> and worker processes can use it?
> >>
> >> Context :
> >>
> >> I am working on a DAG that involves making an Oracle connection, which
> >> requires the LD_LIBRARY_PATH variable to be set. I added it to
> >> /etc/profile.d of the airflow user. When I start a new shell + python
> >> virtualenv and run “airflow test dag_name task_name”, the task is able
> to
> >> use the path variable above and make the necessary connection.
> >>
> >> However, when it runs through the scheduler, the task fails with the
> >> following error : “Oracle client library cannot be loaded”, presumably
> >> because it cannot read the LD_LIBRARY_PATH.
> >>
> >> 2) I have also tried the following code in my custom hook, to explicitly
> >> set the path. I see the same error.
> >>
> >> if 'LD_LIBRARY_PATH' not in os.environ.keys():
> >>    os.environ['LD_LIBRARY_PATH']="/opt/oracle/instantclient_11_2"
> >>    logging.info("added ld library path", os.environ['LD_LIBRARY_PATH'])
> >>
> >> Any pointers on how I can resolve this?
> >>
> >> Thanks,
> >> Rajesh
> >>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message