airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Greg Reda (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-240) TemplateNotFound Error when using PostgresOperator and SQL script
Date Tue, 14 Jun 2016 21:10:31 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Greg Reda updated AIRFLOW-240:
------------------------------
    Description: 
When using the PostgresOperator, I've been unable to get templating for instance. Here's an
example to reproduce the error:

{code:title=/my_dag/csv_to_redshift.py}
copy_s3_to_redshift = PostgresOperator(
    task_id='load_table',
    sql=this_dag_path + '/copy_to_redshift.sql',
    params=dict(
        AWS_ACCESS_KEY_ID=Variable.get('AWS_ACCESS_KEY_ID'),
        AWS_SECRET_ACCESS_KEY=Variable.get('AWS_SECRET_ACCESS_KEY')
    ),
    postgres_conn_id='postgres_redshift',
    autocommit=False,
    dag=dag
)
{code}

{code:title=/my_dag/copy_to_redshift.sql}
COPY public.table_foobar FROM 's3://mybucket/test-data/import/foobar.csv'
CREDENTIALS 'aws_access_key_id={{ AWS_ACCESS_KEY_ID }};aws_secret_access_key={{ AWS_SECRET_ACCESS_KEY
}}'
CSV
NULL as 'null'
IGNOREHEADER as 1;
{code}

{code}
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 2038, in resolve_template_files
    setattr(self, attr, env.loader.get_source(env, content)[0])
  File "/usr/local/lib/python2.7/dist-packages/jinja2/loaders.py", line 187, in get_source
    raise TemplateNotFound(template)
TemplateNotFound: /home/gregreda/airflow/dags/example_csv_to_redshift/copy_to_redshift.sql
{code}

Any ideas?

  was:
When using the PostgresOperator, I've been unable to get templating for instance. Here's an
example to reproduce the error:

/my_dag/csv_to_redshift.py
{code:title=/my_dag/csv_to_redshift.py}
copy_s3_to_redshift = PostgresOperator(
    task_id='load_table',
    sql=this_dag_path + '/copy_to_redshift.sql',
    params=dict(
        AWS_ACCESS_KEY_ID=Variable.get('AWS_ACCESS_KEY_ID'),
        AWS_SECRET_ACCESS_KEY=Variable.get('AWS_SECRET_ACCESS_KEY')
    ),
    postgres_conn_id='postgres_redshift',
    autocommit=False,
    dag=dag
)
{code}

{code:title=/my_dag/copy_to_redshift.sql}
COPY public.table_foobar FROM 's3://mybucket/test-data/import/foobar.csv'
CREDENTIALS 'aws_access_key_id={{ AWS_ACCESS_KEY_ID }};aws_secret_access_key={{ AWS_SECRET_ACCESS_KEY
}}'
CSV
NULL as 'null'
IGNOREHEADER as 1;
{code}

{code}
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 2038, in resolve_template_files
    setattr(self, attr, env.loader.get_source(env, content)[0])
  File "/usr/local/lib/python2.7/dist-packages/jinja2/loaders.py", line 187, in get_source
    raise TemplateNotFound(template)
TemplateNotFound: /home/gregreda/airflow/dags/example_csv_to_redshift/copy_to_redshift.sql
{code}

Any ideas?


> TemplateNotFound Error when using PostgresOperator and SQL script
> -----------------------------------------------------------------
>
>                 Key: AIRFLOW-240
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-240
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: operators
>         Environment: Ubuntu 14.04
>            Reporter: Greg Reda
>
> When using the PostgresOperator, I've been unable to get templating for instance. Here's
an example to reproduce the error:
> {code:title=/my_dag/csv_to_redshift.py}
> copy_s3_to_redshift = PostgresOperator(
>     task_id='load_table',
>     sql=this_dag_path + '/copy_to_redshift.sql',
>     params=dict(
>         AWS_ACCESS_KEY_ID=Variable.get('AWS_ACCESS_KEY_ID'),
>         AWS_SECRET_ACCESS_KEY=Variable.get('AWS_SECRET_ACCESS_KEY')
>     ),
>     postgres_conn_id='postgres_redshift',
>     autocommit=False,
>     dag=dag
> )
> {code}
> {code:title=/my_dag/copy_to_redshift.sql}
> COPY public.table_foobar FROM 's3://mybucket/test-data/import/foobar.csv'
> CREDENTIALS 'aws_access_key_id={{ AWS_ACCESS_KEY_ID }};aws_secret_access_key={{ AWS_SECRET_ACCESS_KEY
}}'
> CSV
> NULL as 'null'
> IGNOREHEADER as 1;
> {code}
> {code}
> Traceback (most recent call last):
>   File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 2038, in resolve_template_files
>     setattr(self, attr, env.loader.get_source(env, content)[0])
>   File "/usr/local/lib/python2.7/dist-packages/jinja2/loaders.py", line 187, in get_source
>     raise TemplateNotFound(template)
> TemplateNotFound: /home/gregreda/airflow/dags/example_csv_to_redshift/copy_to_redshift.sql
> {code}
> Any ideas?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message