beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Forman-Gornall (JIRA)" <>
Subject [jira] [Created] (BEAM-1908) Allow setting CREATE_NEVER when using a tablespec in BigQueryIO
Date Fri, 07 Apr 2017 18:17:41 GMT
Josh Forman-Gornall created BEAM-1908:

             Summary: Allow setting CREATE_NEVER when using a tablespec in BigQueryIO
                 Key: BEAM-1908
             Project: Beam
          Issue Type: Improvement
          Components: sdk-java-extensions
    Affects Versions: 0.6.0
            Reporter: Josh Forman-Gornall
            Assignee: Davor Bonaci
            Priority: Trivial

Currently, the BigQueryIO.Write transform doesn't allow CreateDisposition.CREATE_NEVER to
be set when using a tablespec to determine the BigQuery table name dynamically at runtime.

This check was originally put in because it was assumed that BigQueryIO would need to create
a new table every time the tablespec returns a table name that hasn't been seen before. Since
then a new BigQuery feature was released which enables date partitioning within a single table,
and is now the preferred way to shard data by date (instead of having a table per date). It
should therefore be possible to use tablespec to write to a specific partition e.g. `my-project:dataset.my_table$20170407`,
while setting CreateDisposition.CREATE_NEVER since we never need to create a new table.

This message was sent by Atlassian JIRA

View raw message