Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 2480D200D52 for ; Sat, 18 Nov 2017 03:12:28 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 22FFA160C0B; Sat, 18 Nov 2017 02:12:28 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 69624160BFB for ; Sat, 18 Nov 2017 03:12:27 +0100 (CET) Received: (qmail 25996 invoked by uid 500); 18 Nov 2017 02:12:26 -0000 Mailing-List: contact commits-help@beam.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@beam.apache.org Delivered-To: mailing list commits@beam.apache.org Received: (qmail 25987 invoked by uid 99); 18 Nov 2017 02:12:26 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Nov 2017 02:12:26 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 78D82F551E; Sat, 18 Nov 2017 02:12:25 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: chamikara@apache.org To: commits@beam.apache.org Date: Sat, 18 Nov 2017 02:12:25 -0000 Message-Id: <1a22a54b668c4455bae0223a85dc7e2c@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [1/2] beam git commit: Clarify documentation on support for reading from/writing to time partitioned BQ tables. archived-at: Sat, 18 Nov 2017 02:12:28 -0000 Repository: beam Updated Branches: refs/heads/master e3c1fe4b1 -> 022e74460 Clarify documentation on support for reading from/writing to time partitioned BQ tables. Project: http://git-wip-us.apache.org/repos/asf/beam/repo Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2034c427 Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2034c427 Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2034c427 Branch: refs/heads/master Commit: 2034c4276f3a137a16228ca9a30f84ddd7dd5fd4 Parents: e3c1fe4 Author: chamikara@google.com Authored: Fri Nov 17 15:29:57 2017 -0800 Committer: chamikara@google.com Committed: Fri Nov 17 18:12:01 2017 -0800 ---------------------------------------------------------------------- sdks/python/apache_beam/io/gcp/bigquery.py | 10 ++++++++++ 1 file changed, 10 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/beam/blob/2034c427/sdks/python/apache_beam/io/gcp/bigquery.py ---------------------------------------------------------------------- diff --git a/sdks/python/apache_beam/io/gcp/bigquery.py b/sdks/python/apache_beam/io/gcp/bigquery.py index ee79ae5..a79ad5e 100644 --- a/sdks/python/apache_beam/io/gcp/bigquery.py +++ b/sdks/python/apache_beam/io/gcp/bigquery.py @@ -78,6 +78,16 @@ When creating a BigQuery input transform, users should provide either a query or a table. Pipeline construction will fail with a validation error if neither or both are specified. +**Time partitioned tables** + +BigQuery sink currently does not fully support writing to BigQuery +time partitioned tables. But writing to a *single* partition may work if +that does not involve creating a new table (for example, when writing to an +existing table with `create_disposition=CREATE_NEVER` and +`write_disposition=WRITE_APPEND`). +BigQuery source supports reading from a single time partition with the partition +decorator specified as a part of the table identifier. + *** Short introduction to BigQuery concepts *** Tables have rows (TableRow) and each row has cells (TableCell). A table has a schema (TableSchema), which in turn describes the schema of each