beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chamikara Jayalath (JIRA)" <>
Subject [jira] [Commented] (BEAM-2673) BigQuery Sink should use the Load API
Date Mon, 24 Jul 2017 20:30:00 GMT


Chamikara Jayalath commented on BEAM-2673:

I think the fix here is to add a new BQ sink (which will work for both Direct and Dataflow
runners) which will be a considerably large change. I agree that we should prioritize this
but not sure if 2.2.0 is viable.

Also, IIUC, currently the delay only gets hit at DirectRunner when using WRITE_TRUNCATE which
is not that severe.

> BigQuery Sink should use the Load API
> -------------------------------------
>                 Key: BEAM-2673
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-py
>            Reporter: Sourabh Bajaj
>            Assignee: Ahmet Altay
> Currently the BigQuery sink is written to by using the streaming api in the direct runner.
Instead we should just use the load api and also simplify the management of different create
and write disposition. 

This message was sent by Atlassian JIRA

View raw message