beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kevin Peterson (JIRA)" <>
Subject [jira] [Commented] (BEAM-1834) Bigquery Write validation doesn't work well with ValueInSingleWindow
Date Wed, 29 Mar 2017 22:46:41 GMT


Kevin Peterson commented on BEAM-1834:

> The latter is explicitly not supported at this time.

Support for data-dependent tables was recently added on master:

> Bigquery Write validation doesn't work well with ValueInSingleWindow
> --------------------------------------------------------------------
>                 Key: BEAM-1834
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-gcp
>            Reporter: Kevin Peterson
> I am using the new {{Write<T> to(SerializableFunction<ValueInSingleWindow<T>,
String> tableSpecFunction)}} function to write data to different Bigquery tables depending
on the values. I'm my case, the values can have a different schema (it starts as an {{Any}}
encoded protobuf, which I parse and expand to a {{TableRow}} object).
> Since the tables have different schemas, the existing implementation of {{withSchema}}
doesn't work.
> Some options:
> # Allow {{CreateDisposition.CREATE_NEVER}} in this situation. Failed inserts from a missing
table just fail (and eventually pass through via BEAM-190).
> # Add a new {{withSchema(SerializableFunction<ValueInSingleWindow<T>, TableSchema>}}
> I think eventually both of the above should be allowable configurations, but just one
will unblock my current error. Happy to implement, given some guidance on design preferences.

This message was sent by Atlassian JIRA

View raw message