airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jack (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AIRFLOW-3254) BigQueryGetDataOperator to support reading query from SQL file
Date Thu, 25 Oct 2018 05:33:00 GMT
jack created AIRFLOW-3254:
-----------------------------

             Summary: BigQueryGetDataOperator to support reading query from SQL file
                 Key: AIRFLOW-3254
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3254
             Project: Apache Airflow
          Issue Type: Improvement
    Affects Versions: 1.10.0
            Reporter: jack


As discussed with [~Fokko] on Slack:

Currently the BigQueryGetDataOperator supports only reading query provided directly as:

 
{code:java}
sql = 'SELECT ID FROM TABLE'
{code}
 

it does not support reading the query from a SQL file which can be annoying as sometimes queries
are quite large.

This behavior is supported by other operators like MySqlToGoogleCloudStorageOperator:

dag = DAG(
    dag_id='Import',
    default_args=args,
    schedule_interval='*/5 * * * *',
    max_active_runs=1,
    catchup=False,
    template_searchpath = ['/home/.../airflow/…/sql/Import']
)

 

importop = MySqlToGoogleCloudStorageOperator(
    task_id='import',
    mysql_conn_id='MySQL_con',
    google_cloud_storage_conn_id='gcp_con',
    provide_context=True,
    sql = 'importop.sql',
    params=\{'table_name' : TABLE_NAME},
    bucket=GCS_BUCKET_ID,
    filename=file_name_orders,
    dag=dag)

 

If anyone can pick it up it would be great :)

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message