Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
From: Rajesh Chamarthi <rajesh.chamarthi@gmail.com>
Content-Type: text/plain;
	charset=utf-8
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (Mac OS X Mail 11.0 \(3445.1.7\))
Subject: Re: Environment variables - Oracle LD_LIBRARY_PATH
Date: Fri, 8 Dec 2017 19:32:37 -0500
References: <77E58FF4-5A1C-4472-9018-C97E0A4F7295@gmail.com>
 <CAKMdtFdTaXWVuCycjx6OUVoiLKUApJoe343N67X9UJ9a282TZg@mail.gmail.com>
To: dev@airflow.incubator.apache.org
In-Reply-To: <CAKMdtFdTaXWVuCycjx6OUVoiLKUApJoe343N67X9UJ9a282TZg@mail.gmail.com>
Message-Id: <E14482A7-0B36-473A-A090-5BBF446B1D78@gmail.com>
archived-at: Sat, 09 Dec 2017 00:32:51 -0000

Hi Joy,

Thank you, =E2=80=9Cairflow test=E2=80=9D does not seem like a good way =
to test environment in that case :)

Yes, I am running using celery executor and running the tasks in a =
distributed manner. I currently set the environment variable in the hook =
just before it tries to get a DB connection.

We also use supervisord to manage the worker, scheduler and web server =
processes and I have changed the worker=E2=80=99s env profile to set the =
environment variable, but it does not work either.=20


> On Dec 8, 2017, at 2:50 PM, Joy Gao <joyg@wepay.com> wrote:
>=20
> Hi Rajesh,
>=20
> `airflow test` executes the task synchronously in the same python =
process
> (without touching the scheduler), which is probably why it went down =
the
> happy path.
>=20
> Are you using celery executor and running the tasks in a distributed
> manner? It* sounds like* the task execution env doesn't know about the
> variable.
>=20
> Also depending on where you added the code to your custom hook, it may =
be
> in a different process from which the task is executed, since =
os.environ
> sets variable only for the scope/duration of the python process (and =
its
> children).
>=20
>=20
> On Fri, Dec 8, 2017 at 7:55 AM, Rajesh Chamarthi =
<rajesh.chamarthi@gmail.com
>> wrote:
>=20
>> Hi all,
>>=20
>> What is the right way to set environment variables to be used in =
DAGS.
>> Also, is it possible to export them such that currently running =
scheduler
>> and worker processes can use it?
>>=20
>> Context :
>>=20
>> I am working on a DAG that involves making an Oracle connection, =
which
>> requires the LD_LIBRARY_PATH variable to be set. I added it to
>> /etc/profile.d of the airflow user. When I start a new shell + python
>> virtualenv and run =E2=80=9Cairflow test dag_name task_name=E2=80=9D, =
the task is able to
>> use the path variable above and make the necessary connection.
>>=20
>> However, when it runs through the scheduler, the task fails with the
>> following error : =E2=80=9COracle client library cannot be loaded=E2=80=
=9D, presumably
>> because it cannot read the LD_LIBRARY_PATH.
>>=20
>> 2) I have also tried the following code in my custom hook, to =
explicitly
>> set the path. I see the same error.
>>=20
>> if 'LD_LIBRARY_PATH' not in os.environ.keys():
>>    os.environ['LD_LIBRARY_PATH']=3D"/opt/oracle/instantclient_11_2"
>>    logging.info("added ld library path", =
os.environ['LD_LIBRARY_PATH'])
>>=20
>> Any pointers on how I can resolve this?
>>=20
>> Thanks,
>> Rajesh
>>=20
>>=20