beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Halperin (JIRA)" <>
Subject [jira] [Commented] (BEAM-1834) Bigquery Write validation doesn't work well with ValueInSingleWindow
Date Wed, 29 Mar 2017 22:41:41 GMT


Daniel Halperin commented on BEAM-1834:

It sounds like this a feature request for BigQueryIO supporting Data-dependent Schemas in
addition to Data-dependent tables?

The latter is explicitly not supported at this time.

> Bigquery Write validation doesn't work well with ValueInSingleWindow
> --------------------------------------------------------------------
>                 Key: BEAM-1834
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-gcp
>            Reporter: Kevin Peterson
>            Assignee: Daniel Halperin
> I am using the new {{Write<T> to(SerializableFunction<ValueInSingleWindow<T>,
String> tableSpecFunction)}} function to write data to different Bigquery tables depending
on the values. I'm my case, the values can have a different schema (it starts as an {{Any}}
encoded protobuf, which I parse and expand to a {{TableRow}} object).
> Since the tables have different schemas, the existing implementation of {{withSchema}}
doesn't work.
> Some options:
> # Allow {{CreateDisposition.CREATE_NEVER}} in this situation. Failed inserts from a missing
table just fail (and eventually pass through via BEAM-190).
> # Add a new {{withSchema(SerializableFunction<ValueInSingleWindow<T>, TableSchema>}}
> I think eventually both of the above should be allowable configurations, but just one
will unblock my current error. Happy to implement, given some guidance on design preferences.

This message was sent by Atlassian JIRA

View raw message