beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject [beam] branch master updated: withNumFileShards must be used when using withTriggeringFrequency
Date Tue, 30 Apr 2019 23:36:22 GMT
This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a commit to branch master
in repository

The following commit(s) were added to refs/heads/master by this push:
     new 73be1e3  withNumFileShards must be used when using withTriggeringFrequency
     new fdf84ab  Merge pull request #8244 from ttanay/with-num-file-shards
73be1e3 is described below

commit 73be1e395addf14c6473326e9c70a808eb086da2
Author: ttanay <>
AuthorDate: Sat Apr 6 20:53:21 2019 +0530

    withNumFileShards must be used when using withTriggeringFrequency
    The default value of `numFileShards` is 0 for file loads. When
    writing to BigQuery using file loads in streaming, if `numFileShards`
    is 0, it throws an Exception. Therefore, `withNumFileShards` must
    be used along with `withTriggeringFrequency` to ensure `numFileShards`
    is not 0.
 .../src/main/java/org/apache/beam/sdk/io/gcp/bigquery/   | 2 +-
 website/src/documentation/io/                | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/
index 85cdba4..5dd6cb9 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/
@@ -1814,7 +1814,7 @@ public class BigQueryIO {
      * Control how many file shards are written when using BigQuery load jobs. Applicable
only when
-     * also setting {@link #withTriggeringFrequency}. The default value is 1000.
+     * also setting {@link #withTriggeringFrequency}.
     public Write<T> withNumFileShards(int numFileShards) {
diff --git a/website/src/documentation/io/ b/website/src/documentation/io/
index 855b5cf..540647a 100644
--- a/website/src/documentation/io/
+++ b/website/src/documentation/io/
@@ -593,7 +593,7 @@ for the list of the available methods and their restrictions.
 ***Note:*** If you use batch loads in a streaming pipeline, you must use
-`withTriggeringFrequency` to specify a triggering frequency.
+`withTriggeringFrequency` to specify a triggering frequency and `withNumFileShards` to specify
number of file shards written.
 ### Writing to a table

View raw message