beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pabl...@apache.org
Subject [beam] branch master updated: withNumFileShards must be used when using withTriggeringFrequency
Date Tue, 30 Apr 2019 23:36:22 GMT
This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
     new 73be1e3  withNumFileShards must be used when using withTriggeringFrequency
     new fdf84ab  Merge pull request #8244 from ttanay/with-num-file-shards
73be1e3 is described below

commit 73be1e395addf14c6473326e9c70a808eb086da2
Author: ttanay <ttanay100@gmail.com>
AuthorDate: Sat Apr 6 20:53:21 2019 +0530

    withNumFileShards must be used when using withTriggeringFrequency
    
    The default value of `numFileShards` is 0 for file loads. When
    writing to BigQuery using file loads in streaming, if `numFileShards`
    is 0, it throws an Exception. Therefore, `withNumFileShards` must
    be used along with `withTriggeringFrequency` to ensure `numFileShards`
    is not 0.
---
 .../src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java   | 2 +-
 website/src/documentation/io/built-in-google-bigquery.md                | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
index 85cdba4..5dd6cb9 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
@@ -1814,7 +1814,7 @@ public class BigQueryIO {
 
     /**
      * Control how many file shards are written when using BigQuery load jobs. Applicable
only when
-     * also setting {@link #withTriggeringFrequency}. The default value is 1000.
+     * also setting {@link #withTriggeringFrequency}.
      */
     @Experimental
     public Write<T> withNumFileShards(int numFileShards) {
diff --git a/website/src/documentation/io/built-in-google-bigquery.md b/website/src/documentation/io/built-in-google-bigquery.md
index 855b5cf..540647a 100644
--- a/website/src/documentation/io/built-in-google-bigquery.md
+++ b/website/src/documentation/io/built-in-google-bigquery.md
@@ -593,7 +593,7 @@ for the list of the available methods and their restrictions.
 
 {:.language-java}
 ***Note:*** If you use batch loads in a streaming pipeline, you must use
-`withTriggeringFrequency` to specify a triggering frequency.
+`withTriggeringFrequency` to specify a triggering frequency and `withNumFileShards` to specify
number of file shards written.
 
 
 ### Writing to a table


Mime
View raw message