beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Johston (JIRA)" <>
Subject [jira] [Commented] (BEAM-2390) allow user to use .setTimePartitioning in BigQueryIO.write
Date Thu, 01 Jun 2017 04:12:04 GMT


Eric Johston commented on BEAM-2390:

My initial commit had some errors in that

a) TimePartitioning is not serializable, and
b) CreateTables#possibleCreate fails when TableDestination includes a partition with $

I've made changes to these by storing TimePartitioning in Json format similar to how the schemas
are propagated. I've also modified the table creation such that when creating a table Beam
only looks at the part before $. It seems to be working now (running this from my own fork
in production)

> allow user to use .setTimePartitioning in BigQueryIO.write
> ----------------------------------------------------------
>                 Key: BEAM-2390
>                 URL:
>             Project: Beam
>          Issue Type: Improvement
>          Components: beam-model-runner-api
>    Affects Versions: 2.0.0
>            Reporter: Eric Johston
>            Assignee: Kenneth Knowles
>              Labels: easyfix, features, newbie
>             Fix For: 2.0.0
>   Original Estimate: 2h
>  Remaining Estimate: 2h
> Currently when writing to a table with BigQueryIO sink, there is no way to create a new
table that is date partitioned. This would be very useful, since currently the only way to
do this is  by manually creating a table ahead of time. We should be able to leverage the
automatic table creation functionality for date partitioned tables.
> The best way to do this would be to have a withTimePartitioning method in the BigQueryIO

This message was sent by Atlassian JIRA

View raw message