airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (Jira)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-5542) Export a BigQuery query result to GCS
Date Wed, 02 Oct 2019 13:41:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-5542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16942823#comment-16942823
] 

ASF GitHub Bot commented on AIRFLOW-5542:
-----------------------------------------

meftasadat commented on pull request #6173: [AIRFLOW-5542] Added BQ query results export to
GCS  operator
URL: https://github.com/apache/airflow/pull/6173
 
 
   
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Export a BigQuery query result to GCS
> -------------------------------------
>
>                 Key: AIRFLOW-5542
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5542
>             Project: Apache Airflow
>          Issue Type: New Feature
>          Components: operators
>    Affects Versions: 1.10.5
>            Reporter: Mefta Sadat
>            Priority: Minor
>
> h2. Problem
> There is currently no direct operator to export a BQ query result dataset to GCS.
> h2. Proposed implementation
> This problem can be solved by creating a temporary table, storing the query results in
the temp table, exporting the table to GCS and then finally deleting the table
> The following methods in airflow.gcp.hooks.bigquery_hook can be leveraged to achieve
this solution:
> create_empty_dataset -> run_query -> run_extract -> run_table_delete -> delete_dataset
> The proposer operator is just a wrapper around these methods. It executes the above-mentioned
methods in the necessary order and handles any exceptions that may occur.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message