airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rajesh Chamarthi <rajesh.chamar...@gmail.com>
Subject Re: Environment variables - Oracle LD_LIBRARY_PATH
Date Fri, 15 Dec 2017 03:24:50 GMT
 Will give this a try. I tried the airflow worker’s conf and it did d not
work.

I fixed my problem (temporarily) by setting the environment variable in the
python __init__ code before it calls the database connection create
statement. (Using python os.environ).
On Mon, Dec 11, 2017 at 9:50 AM William Pursell <williamp@wepay.com> wrote:

> It's perhaps not the cleanest solution, but I have had success setting the
> environment in the supervisord config file.  Eg: add the line
>
> environment=LD_LIBRARY_PATH="/opt/oracle/instantclient_11_2"
>
> On Fri, Dec 8, 2017 at 4:32 PM, Rajesh Chamarthi <
> rajesh.chamarthi@gmail.com
> > wrote:
>
> > Hi Joy,
> >
> > Thank you, “airflow test” does not seem like a good way to test
> > environment in that case :)
> >
> > Yes, I am running using celery executor and running the tasks in a
> > distributed manner. I currently set the environment variable in the hook
> > just before it tries to get a DB connection.
> >
> > We also use supervisord to manage the worker, scheduler and web server
> > processes and I have changed the worker’s env profile to set the
> > environment variable, but it does not work either.
> >
> >
> > > On Dec 8, 2017, at 2:50 PM, Joy Gao <joyg@wepay.com> wrote:
> > >
> > > Hi Rajesh,
> > >
> > > `airflow test` executes the task synchronously in the same python
> process
> > > (without touching the scheduler), which is probably why it went down
> the
> > > happy path.
> > >
> > > Are you using celery executor and running the tasks in a distributed
> > > manner? It* sounds like* the task execution env doesn't know about the
> > > variable.
> > >
> > > Also depending on where you added the code to your custom hook, it may
> be
> > > in a different process from which the task is executed, since
> os.environ
> > > sets variable only for the scope/duration of the python process (and
> its
> > > children).
> > >
> > >
> > > On Fri, Dec 8, 2017 at 7:55 AM, Rajesh Chamarthi <
> > rajesh.chamarthi@gmail.com
> > >> wrote:
> > >
> > >> Hi all,
> > >>
> > >> What is the right way to set environment variables to be used in DAGS.
> > >> Also, is it possible to export them such that currently running
> > scheduler
> > >> and worker processes can use it?
> > >>
> > >> Context :
> > >>
> > >> I am working on a DAG that involves making an Oracle connection, which
> > >> requires the LD_LIBRARY_PATH variable to be set. I added it to
> > >> /etc/profile.d of the airflow user. When I start a new shell + python
> > >> virtualenv and run “airflow test dag_name task_name”, the task is able
> > to
> > >> use the path variable above and make the necessary connection.
> > >>
> > >> However, when it runs through the scheduler, the task fails with the
> > >> following error : “Oracle client library cannot be loaded”, presumably
> > >> because it cannot read the LD_LIBRARY_PATH.
> > >>
> > >> 2) I have also tried the following code in my custom hook, to
> explicitly
> > >> set the path. I see the same error.
> > >>
> > >> if 'LD_LIBRARY_PATH' not in os.environ.keys():
> > >>    os.environ['LD_LIBRARY_PATH']="/opt/oracle/instantclient_11_2"
> > >>    logging.info("added ld library path",
> os.environ['LD_LIBRARY_PATH'])
> > >>
> > >> Any pointers on how I can resolve this?
> > >>
> > >> Thanks,
> > >> Rajesh
> > >>
> > >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message