beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andre (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-2370) BigQuery Insert with Partition Decorator throwing error
Date Mon, 05 Jun 2017 09:14:04 GMT

    [ https://issues.apache.org/jira/browse/BEAM-2370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16036702#comment-16036702
] 

Andre commented on BEAM-2370:
-----------------------------

Any idea on this?

> BigQuery Insert with Partition Decorator throwing error
> -------------------------------------------------------
>
>                 Key: BEAM-2370
>                 URL: https://issues.apache.org/jira/browse/BEAM-2370
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-gcp
>    Affects Versions: 2.0.0
>         Environment: DirectRunner
>            Reporter: Andre
>            Assignee: Reuven Lax
>
> Running a DataFlow job with the DirectRunner which is inserting data into a partitioned
table using decorators throws the following error multiple times BUT still inserts records
into the right partition.
> {code:java|title=Error}
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl executeWithRetries
> INFO: Ignore the error and retry the request.
> com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request
> {
>   "code" : 400,
>   "errors" : [ {
>     "domain" : "global",
>     "message" : "Invalid table ID \"mytable_orders$20170516\". Table IDs must be alphanumeric
(plus underscores) and must be at most 1024 characters long. Also, Table decorators cannot
be used.",
>     "reason" : "invalid"
>   } ],
>   "message" : "Invalid table ID \"mytable_orders$20170516\". Table IDs must be alphanumeric
(plus underscores) and must be at most 1024 characters long. Also, Table decorators cannot
be used."
> }
> 	at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
> {code}
> {code:java|title=Code}
> // Write TableRows to BQ
> rows.apply("TransformationStep", ParDo.of(new Outputter()))
>  .apply("WindowDaily", Window. <TableRow> into(CalendarWindows.days(1)))
>  .apply("WriteToBQ", BigQueryIO.writeTableRows()
>   .to(new SerializableFunction <ValueInSingleWindow<TableRow> , TableDestination>
() {
>    private static final long serialVersionUID = 8196602721734820219 L;
>    @Override
>    public TableDestination apply(ValueInSingleWindow <TableRow> value) {
>     String dayString = DateTimeFormat.forPattern("yyyyMMdd").withZone(DateTimeZone.UTC)
>      .print(((IntervalWindow) value.getWindow()).start());
>     TableDestination td = new TableDestination("my-project:dataset.mytable_orders$" +
dayString, "");
>     return td;
>    }
>   }).withSchema(mySchema)
>   .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_NEVER)
>   .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_TRUNCATE));
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message