beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <>
Subject [jira] [Work logged] (BEAM-2817) Bigquery queries should allow options to run in batch mode or not
Date Wed, 14 Mar 2018 21:14:00 GMT


ASF GitHub Bot logged work on BEAM-2817:

                Author: ASF GitHub Bot
            Created on: 14/Mar/18 21:13
            Start Date: 14/Mar/18 21:13
    Worklog Time Spent: 10m 
      Work Description: jkff commented on a change in pull request #4606: [BEAM-2817] BigQuery
queries are allowed to run in either BATCH or IN…

 File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/
 @@ -74,12 +77,19 @@ private BigQueryQuerySource(
       Boolean useLegacySql,
       BigQueryServices bqServices,
       Coder<T> coder,
-      SerializableFunction<SchemaAndRecord, T> parseFn) {
+      SerializableFunction<SchemaAndRecord, T> parseFn,
+      Priority priority) {
     super(stepUuid, bqServices, coder, parseFn);
     this.query = checkNotNull(query, "query");
     this.flattenResults = checkNotNull(flattenResults, "flattenResults");
     this.useLegacySql = checkNotNull(useLegacySql, "useLegacySql");
     this.dryRunJobStats = new AtomicReference<>();
+    if (priority != BigQueryIO.TypedRead.Priority.BATCH
 Review comment:
   This condition is always true - "priority != a || priority != b" is true by definition,
because it can't equal a and b at the same time. I suppose you probably meant "&&",
but then again I'm not sure why this condition is needed at all - why not just "this.priority
= priority"?

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:

Issue Time Tracking

    Worklog Id:     (was: 80563)
    Time Spent: 2h 10m  (was: 2h)

> Bigquery queries should allow options to run in batch mode or not
> -----------------------------------------------------------------
>                 Key: BEAM-2817
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: io-java-gcp
>    Affects Versions: 2.0.0
>            Reporter: Lara Schmidt
>            Assignee: Justin Tumale
>            Priority: Major
>              Labels: newbie, starter
>             Fix For: 2.4.0
>          Time Spent: 2h 10m
>  Remaining Estimate: 0h
> When bigquery read does a query it sets the mode to batch. A batch query can be very
slow to schedule as it batches it with other queries. However it doesn't use batch quota which
is better for some cases. However, in some cases a fast query is better (especially in timed
tests). It would be a good idea to have a configuration to the bigquery source to set this

This message was sent by Atlassian JIRA

View raw message