airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nathan Scully (JIRA)" <>
Subject [jira] [Created] (AIRFLOW-253) Job does not find CONN_ID environmental variable
Date Fri, 17 Jun 2016 06:36:05 GMT
Nathan Scully created AIRFLOW-253:

             Summary: Job does not find CONN_ID environmental variable
                 Key: AIRFLOW-253
             Project: Apache Airflow
          Issue Type: Bug
          Components: hooks
    Affects Versions:
         Environment: OS: Ubuntu 14.04
airflow@ip-10-0-40-185:~$ pip show airflow
Name: airflow
            Reporter: Nathan Scully

I am not sure if this is directly an airflow issue or something causing havoc in my environment
but when the scheduler triggers a test job I have setup (@hourly) to ping a Redshift server
with a 'select * from x limit 1;' I receive a no conn_id found. However, I can echo the environmental
var and if I run the job as a test, or manually query from python console I can resolve it.
Any ideas what might be causing it to be missed? 

Error from AIRFLOW scheduled run:
    [2016-06-17 06:00:04,402] {} ERROR - The conn_id `OF_REDSHIFT` isn't defined
    Traceback (most recent call last):
      File "/usr/local/lib/python2.7/dist-packages/airflow/", line 1245, in run
        result = task_copy.execute(context=context)
      File "/usr/local/lib/python2.7/dist-packages/airflow/operators/",
line 66, in execute
        return_value = self.python_callable(*self.op_args, **self.op_kwargs)
      File "/usr/local/airflow/dags/", line 23, in ping_redshift_query
        df = ph.get_records(sql)
      File "/usr/local/lib/python2.7/dist-packages/airflow/hooks/", line 72,
in get_records
        conn = self.get_conn()
      File "/usr/local/lib/python2.7/dist-packages/airflow/hooks/", line 18,
in get_conn
        conn = self.get_connection(self.postgres_conn_id)
      File "/usr/local/lib/python2.7/dist-packages/airflow/hooks/", line 51, in
        conn = random.choice(cls.get_connections(conn_id))
      File "/usr/local/lib/python2.7/dist-packages/airflow/hooks/", line 39, in
        "The conn_id `{0}` isn't defined".format(conn_id))  
    AirflowException: The conn_id `OF_REDSHIFT` isn't defined
    [2016-06-17 06:00:04,403] {} INFO - Marking task as FAILED.
    [2016-06-17 06:00:04,695] {} ERROR - The conn_id `OF_REDSHIFT` isn't defined
I can echo the env variable:
    airflow@ip-xx-x-xx-xxx:~$ echo $AIRFLOW_CONN_OF_REDSHIFT

And can also get it through python console:
    airflow@ip-xx-x-xx-xxx:~$ python
    Python 2.7.6 (default, Jun 22 2015, 17:58:13) 
    [GCC 4.8.2] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import os
    >>> conn_id ='OF_REDSHIFT'
    >>> os.environ.get(CONN_ENV_PREFIX + conn_id.upper())

This message was sent by Atlassian JIRA

View raw message