airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Akshar Dave (JIRA)" <j...@apache.org>
Subject [jira] [Issue Comment Deleted] (AIRFLOW-3114) Add feature to create External BigQuery Table for querying Cloud Bigtable data
Date Tue, 25 Sep 2018 22:39:00 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-3114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Akshar Dave updated AIRFLOW-3114:
---------------------------------
    Comment: was deleted

(was: This can probably be implemented by adding new BigQueryCreateExternalBigTableOperator
to the current BigQuery Operator. )

> Add feature to create External BigQuery Table for querying Cloud Bigtable data
> ------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-3114
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3114
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: contrib, gcp, hooks
>    Affects Versions: 1.10.0, 2.0.0
>            Reporter: Akshar Dave
>            Priority: Minor
>             Fix For: 2.0.0, 1.10.1
>
>
> Currently using the BigQuery operator and the bq hook we can create an external BQ table
but there is no way to create a permanent external table on Cloud Bigtable data.
>  * For [externalDataConfiguration.sourceUris|https://cloud.google.com/bigquery/docs/reference/v2/tables#externalDataConfiguration.sourceUris],
you can specify exactly one [Cloud Bigtable URI|https://cloud.google.com/bigquery/external-data-bigtable#bigtable-uri],
and it must be a fully specified, valid HTTPS URL.
>  * Specify the data format properties by setting the [externalDataConfiguration.sourceFormat|https://cloud.google.com/bigquery/docs/reference/v2/tables#externalDataConfiguration.sourceFormat] property.
For Google Cloud Bigtable, specify {{"BIGTABLE"}}.
> Below are the links to Doc, Python Api and the JSON Rest API that can be helpful:
> [https://cloud.google.com/bigquery/external-data-bigtable]
> [https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#externalDataConfiguration.sourceFormat]
> [https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#externalDataConfiguration.sourceUris]
> [https://github.com/GoogleCloudPlatform/google-cloud-python/blob/38080c8920d20b01f06c1eaef79eb527525de552/bigquery/google/cloud/bigquery/external_config.py]
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message