beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eugene Kirpichov (JIRA)" <>
Subject [jira] [Closed] (BEAM-1834) Bigquery Write validation doesn't work well with ValueInSingleWindow
Date Mon, 11 Dec 2017 17:38:05 GMT


Eugene Kirpichov closed BEAM-1834.
       Resolution: Fixed
    Fix Version/s: 2.2.0

Support for data-dependent schemas has been added a pretty long time ago via BigQueryIO.write().to(DynamicDestinations).

> Bigquery Write validation doesn't work well with ValueInSingleWindow
> --------------------------------------------------------------------
>                 Key: BEAM-1834
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-gcp
>            Reporter: Kevin Peterson
>             Fix For: 2.2.0
> I am using the new {{Write<T> to(SerializableFunction<ValueInSingleWindow<T>,
String> tableSpecFunction)}} function to write data to different Bigquery tables depending
on the values. I'm my case, the values can have a different schema (it starts as an {{Any}}
encoded protobuf, which I parse and expand to a {{TableRow}} object).
> Since the tables have different schemas, the existing implementation of {{withSchema}}
doesn't work.
> Some options:
> # Allow {{CreateDisposition.CREATE_NEVER}} in this situation. Failed inserts from a missing
table just fail (and eventually pass through via BEAM-190).
> # Add a new {{withSchema(SerializableFunction<ValueInSingleWindow<T>, TableSchema>}}
> I think eventually both of the above should be allowable configurations, but just one
will unblock my current error. Happy to implement, given some guidance on design preferences.

This message was sent by Atlassian JIRA

View raw message