beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Darshan Mehta (JIRA)" <j...@apache.org>
Subject [jira] [Created] (BEAM-3501) BigQuery Partitioned table creation/write fails when destination has partition decorator
Date Fri, 19 Jan 2018 11:28:02 GMT
Darshan Mehta created BEAM-3501:
-----------------------------------

             Summary: BigQuery Partitioned table creation/write fails when destination has
partition decorator
                 Key: BEAM-3501
                 URL: https://issues.apache.org/jira/browse/BEAM-3501
             Project: Beam
          Issue Type: Bug
          Components: beam-model
            Reporter: Darshan Mehta
            Assignee: Kenneth Knowles


Following is the code that writes to BigQuery:

 
{code:java}
BigQueryIO.writeTableRows()
 .to(destination)
 .withCreateDisposition(CREATE_IF_NEEDED)
 .withWriteDisposition(WRITE_APPEND)
 .withSchema(tableSchema)
 .expand(tableRows);{code}
 

Here's the destination's implementation:

 
{code:java}
public TableDestination apply(ValueInSingleWindow<TableRow> input) {
 String partition = timestampExtractor.apply(input.getValue())
 .toString(DateTimeFormat.forPattern("yyyyMMdd").withZoneUTC());
 TableReference tableReference = new TableReference();
 tableReference.setDatasetId(dataset);
 tableReference.setProjectId(projectId);
 tableReference.setTableId(String.format("%s_%s", table, partition));
 log.debug("Will write to BigQuery table: %s", tableReference);
 return new TableDestination(tableReference, null);
}{code}
 

When the dataflow tries to write to this table, I see the following message:
{code:java}
"errors" : [ {
 "domain" : "global",
 "message" : "Cannot read partition information from a table that is not partitioned: <project_id>:<dataset>.<table>$19730522",
 "reason" : "invalid"
 } ]{code}
So, it looks like it's not creating tables with partition in the first place?

 

Apache beam version : 2.2.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message