beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Halperin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-1716) Using Instant type in TableRow should map to the right output format for TIMESTAMP in BigQuery
Date Tue, 25 Apr 2017 15:17:04 GMT

    [ https://issues.apache.org/jira/browse/BEAM-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983049#comment-15983049
] 

Daniel Halperin commented on BEAM-1716:
---------------------------------------

I do not believe directly using a Joda {{Instant}} in {{TableRow}} will work either in Beam
or in the older Dataflow SDK.

Beam uses BigQuery’s streaming insert / load job API, and {{BigQueryIO} just takes whatever
{{TableRow}} object the user provides and serializes it to JSON. My understanding is that
BigQuery does not support importing directly Joda Instant — they expect some form of seconds
since epoch or something like that.

Generally, your code looks good. A few tweaks might be relevant: Do you you want {{/ 1000.0}}?
Do you need to care about time zones?

> Using Instant type in TableRow should map to the right output format for TIMESTAMP in
BigQuery
> ----------------------------------------------------------------------------------------------
>
>                 Key: BEAM-1716
>                 URL: https://issues.apache.org/jira/browse/BEAM-1716
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-gcp
>    Affects Versions: 0.5.0
>            Reporter: Tobias Feldhaus
>              Labels: newbie, starter
>             Fix For: First stable release
>
>
> When using an Instant type in a TableRow, and a TIMESTAMP field in the TableFieldSchema,
one has to convert the Instant via 
> instant.getMillis() / 1000
> to match the BigQuery TIMESTAMP that is "A positive or negative decimal number. A positive
number specifies the number of seconds since the epoch".
> In the Dataflow 1.9 SDK this conversion is done automatically.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message