From commits-return-114386-archive-asf-public=cust-asf.ponee.io@beam.apache.org Fri Jul 9 18:27:06 2021 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mxout1-ec2-va.apache.org (mxout1-ec2-va.apache.org [3.227.148.255]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id 382E7180645 for ; Fri, 9 Jul 2021 20:27:06 +0200 (CEST) Received: from mail.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mxout1-ec2-va.apache.org (ASF Mail Server at mxout1-ec2-va.apache.org) with SMTP id 69A8940652 for ; Fri, 9 Jul 2021 18:27:05 +0000 (UTC) Received: (qmail 99827 invoked by uid 500); 9 Jul 2021 18:27:05 -0000 Mailing-List: contact commits-help@beam.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@beam.apache.org Delivered-To: mailing list commits@beam.apache.org Received: (qmail 99818 invoked by uid 99); 9 Jul 2021 18:27:05 -0000 Received: from ec2-52-202-80-70.compute-1.amazonaws.com (HELO gitbox.apache.org) (52.202.80.70) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2021 18:27:05 +0000 Received: by gitbox.apache.org (ASF Mail Server at gitbox.apache.org, from userid 33) id A2A2F8DC92; Fri, 9 Jul 2021 18:27:04 +0000 (UTC) Date: Fri, 09 Jul 2021 18:26:51 +0000 To: "commits@beam.apache.org" Subject: [beam] branch master updated (0e0aed9 -> 7724cfe) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Message-ID: <162585520738.9618.440279345645203979@gitbox.apache.org> From: apilloud@apache.org X-Git-Host: gitbox.apache.org X-Git-Repo: beam X-Git-Refname: refs/heads/master X-Git-Reftype: branch X-Git-Oldrev: 0e0aed924bee0e2f48f83b5dec6c7c3e3ee2101b X-Git-Newrev: 7724cfe900dd99ba651a62c976b51b486eb69885 X-Git-NotificationType: ref_changed X-Git-Multimail-Version: 1.5.dev Auto-Submitted: auto-generated This is an automated email from the ASF dual-hosted git repository. apilloud pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0e0aed9 [BEAM-12532] correctly handle LocalDateTime and LocalTime with 0 seco… add 7724cfe [BEAM-8933] Add support for reading Arrow over the BigQuery Storage API (#14586) No new revisions were added by this update. Summary of changes: .../org/apache/beam/gradle/BeamModulePlugin.groovy | 4 + .../apache/beam/sdk/schemas/CachingFactory.java | 2 +- .../{sql/perf-tests => arrow}/build.gradle | 24 +- .../beam/sdk/extensions/arrow/ArrowConversion.java | 513 +++++++++++++++ .../beam/sdk/extensions/arrow}/package-info.java | 4 +- .../sdk/extensions/arrow/ArrowConversionTest.java | 176 +++++ .../BigQueryHllSketchCompatibilityIT.java | 2 + sdks/java/io/google-cloud-platform/build.gradle | 4 + .../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 15 + .../gcp/bigquery/BigQueryStorageArrowReader.java | 107 +++ .../io/gcp/bigquery/BigQueryStorageAvroReader.java | 80 +++ .../gcp/bigquery/BigQueryStorageQuerySource.java | 34 +- ...riteRequest.java => BigQueryStorageReader.java} | 29 +- ...uest.java => BigQueryStorageReaderFactory.java} | 25 +- .../io/gcp/bigquery/BigQueryStorageSourceBase.java | 27 +- .../gcp/bigquery/BigQueryStorageStreamSource.java | 31 +- .../gcp/bigquery/BigQueryStorageTableSource.java | 18 +- .../gcp/bigquery/BigQueryIOStorageQueryTest.java | 22 +- .../io/gcp/bigquery/BigQueryIOStorageReadIT.java | 23 +- .../io/gcp/bigquery/BigQueryIOStorageReadTest.java | 723 ++++++++++++++++++++- .../io/gcp/bigquery/BigQueryStorageReaderTest.java | 92 +++ settings.gradle.kts | 1 + 22 files changed, 1862 insertions(+), 94 deletions(-) copy sdks/java/extensions/{sql/perf-tests => arrow}/build.gradle (61%) create mode 100644 sdks/java/extensions/arrow/src/main/java/org/apache/beam/sdk/extensions/arrow/ArrowConversion.java copy {examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/options => sdks/java/extensions/arrow/src/main/java/org/apache/beam/sdk/extensions/arrow}/package-info.java (89%) create mode 100644 sdks/java/extensions/arrow/src/test/java/org/apache/beam/sdk/extensions/arrow/ArrowConversionTest.java create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageArrowReader.java create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageAvroReader.java copy sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/{AvroWriteRequest.java => BigQueryStorageReader.java} (60%) copy sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/{AvroWriteRequest.java => BigQueryStorageReaderFactory.java} (60%) create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageReaderTest.java