airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rajesh Chamarthi <rajesh.chamar...@gmail.com>
Subject Re: Environment variables - Oracle LD_LIBRARY_PATH
Date Sat, 09 Dec 2017 00:32:37 GMT
Hi Joy,

Thank you, “airflow test” does not seem like a good way to test environment in that case
:)

Yes, I am running using celery executor and running the tasks in a distributed manner. I currently
set the environment variable in the hook just before it tries to get a DB connection.

We also use supervisord to manage the worker, scheduler and web server processes and I have
changed the worker’s env profile to set the environment variable, but it does not work either.



> On Dec 8, 2017, at 2:50 PM, Joy Gao <joyg@wepay.com> wrote:
> 
> Hi Rajesh,
> 
> `airflow test` executes the task synchronously in the same python process
> (without touching the scheduler), which is probably why it went down the
> happy path.
> 
> Are you using celery executor and running the tasks in a distributed
> manner? It* sounds like* the task execution env doesn't know about the
> variable.
> 
> Also depending on where you added the code to your custom hook, it may be
> in a different process from which the task is executed, since os.environ
> sets variable only for the scope/duration of the python process (and its
> children).
> 
> 
> On Fri, Dec 8, 2017 at 7:55 AM, Rajesh Chamarthi <rajesh.chamarthi@gmail.com
>> wrote:
> 
>> Hi all,
>> 
>> What is the right way to set environment variables to be used in DAGS.
>> Also, is it possible to export them such that currently running scheduler
>> and worker processes can use it?
>> 
>> Context :
>> 
>> I am working on a DAG that involves making an Oracle connection, which
>> requires the LD_LIBRARY_PATH variable to be set. I added it to
>> /etc/profile.d of the airflow user. When I start a new shell + python
>> virtualenv and run “airflow test dag_name task_name”, the task is able to
>> use the path variable above and make the necessary connection.
>> 
>> However, when it runs through the scheduler, the task fails with the
>> following error : “Oracle client library cannot be loaded”, presumably
>> because it cannot read the LD_LIBRARY_PATH.
>> 
>> 2) I have also tried the following code in my custom hook, to explicitly
>> set the path. I see the same error.
>> 
>> if 'LD_LIBRARY_PATH' not in os.environ.keys():
>>    os.environ['LD_LIBRARY_PATH']="/opt/oracle/instantclient_11_2"
>>    logging.info("added ld library path", os.environ['LD_LIBRARY_PATH'])
>> 
>> Any pointers on how I can resolve this?
>> 
>> Thanks,
>> Rajesh
>> 
>> 


Mime
View raw message