beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Halperin (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (BEAM-1834) Bigquery Write validation doesn't work well with ValueInSingleWindow
Date Wed, 29 Mar 2017 22:41:41 GMT

     [ https://issues.apache.org/jira/browse/BEAM-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Daniel Halperin reassigned BEAM-1834:
-------------------------------------

    Assignee:     (was: Daniel Halperin)

> Bigquery Write validation doesn't work well with ValueInSingleWindow
> --------------------------------------------------------------------
>
>                 Key: BEAM-1834
>                 URL: https://issues.apache.org/jira/browse/BEAM-1834
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-gcp
>            Reporter: Kevin Peterson
>
> I am using the new {{Write<T> to(SerializableFunction<ValueInSingleWindow<T>,
String> tableSpecFunction)}} function to write data to different Bigquery tables depending
on the values. I'm my case, the values can have a different schema (it starts as an {{Any}}
encoded protobuf, which I parse and expand to a {{TableRow}} object).
> Since the tables have different schemas, the existing implementation of {{withSchema}}
doesn't work.
> Some options:
> # Allow {{CreateDisposition.CREATE_NEVER}} in this situation. Failed inserts from a missing
table just fail (and eventually pass through via BEAM-190).
> # Add a new {{withSchema(SerializableFunction<ValueInSingleWindow<T>, TableSchema>}}
function.
> I think eventually both of the above should be allowable configurations, but just one
will unblock my current error. Happy to implement, given some guidance on design preferences.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message