airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-1943) Add feature to create External BigQuery Table
Date Fri, 26 Jan 2018 10:58:02 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16340901#comment-16340901
] 

ASF subversion and git services commented on AIRFLOW-1943:
----------------------------------------------------------

Commit e1bf38942f3fa2fd41f6c4f6824a02b01e92ae38 in incubator-airflow's branch refs/heads/master
from [~kaxilnaik]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=e1bf389 ]

[AIRFLOW-1943] Add External BigQuery Table feature

Add ability to create a BigQuery External Table.
- Add new method create_external_table() in
BigQueryHook()
- Add parameters to existing
GoogleCloudStorageToBigQueryOperator()

Closes #2948 from kaxil/external_table


> Add feature to create External BigQuery Table
> ---------------------------------------------
>
>                 Key: AIRFLOW-1943
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1943
>             Project: Apache Airflow
>          Issue Type: Task
>          Components: contrib, gcp
>    Affects Versions: Airflow 2.0
>            Reporter: Kaxil Naik
>            Assignee: David Sabater
>            Priority: Minor
>             Fix For: 2.0.0
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> Currently using the BigQuery operator and the bq hook we can just create a materialized
(internal) BigQuery Table but there is no code to create an external BQ table.
> Below are the links to Python Api and the JSON Rest API that can be helpful:
> * https://github.com/GoogleCloudPlatform/google-cloud-python/commit/7ea861fce5023409b1f8c0baa1ea7d90b41a2300
> * https://github.com/GoogleCloudPlatform/google-cloud-python/commit/3af22d7fd595e2d16f274d59e34f7a592ae251b5
> * https://cloud.google.com/bigquery/external-table-definition#json_table_definitions_1
> * https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message