beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Work logged] (BEAM-4417) BigqueryIO Numeric datatype Support
Date Tue, 03 Jul 2018 17:54:00 GMT

     [ https://issues.apache.org/jira/browse/BEAM-4417?focusedWorklogId=118787&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-118787
]

ASF GitHub Bot logged work on BEAM-4417:
----------------------------------------

                Author: ASF GitHub Bot
            Created on: 03/Jul/18 17:53
            Start Date: 03/Jul/18 17:53
    Worklog Time Spent: 10m 
      Work Description: ElliottBrossard commented on issue #5755: [BEAM-4417] Support BigQuery's
NUMERIC type using Java
URL: https://github.com/apache/beam/pull/5755#issuecomment-402241675
 
 
   @KumarKishan It looks like you deleted your original post, but I'm guessing the issue is
that this new code expects for the `NUMERIC` column to be represented as a `STRING` in the
Avro file, whereas it is actually `BYTES` with `DECIMAL` logical type. This was the error
that you reported:
   
       java.lang.RuntimeException: org.apache.beam.sdk.util.UserCodeException: com.google.common.base.VerifyException:
Expected Avro schema type STRING, not BYTES, for BigQuery NUMERIC field
   
   The problem is that `AvroCoder` writes `BigDecimal` values as `STRING`s, so I was under
the false impression that this is what `BigQueryAvroUtils` should expect to see :/

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 118787)
    Time Spent: 4h 20m  (was: 4h 10m)

> BigqueryIO Numeric datatype Support
> -----------------------------------
>
>                 Key: BEAM-4417
>                 URL: https://issues.apache.org/jira/browse/BEAM-4417
>             Project: Beam
>          Issue Type: Improvement
>          Components: io-java-gcp
>    Affects Versions: 2.4.0
>            Reporter: Kishan Kumar
>            Assignee: Chamikara Jayalath
>            Priority: Critical
>              Labels: newbie, patch
>             Fix For: 2.6.0
>
>          Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> The BigQueryIO.read fails while parsing the data from the avro file generated while
reading the data from the table which has columns with *Numeric* datatypes. 
> We have gone through the source code at Git-Hub and noticed that *Numeric data type is
not yet supported.* 
>  
> Caused by: com.google.common.base.VerifyException: Unsupported BigQuery type: NUMERIC
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message