From issues-return-148552-archive-asf-public=cust-asf.ponee.io@beam.apache.org Mon Mar 1 06:05:02 2021 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mxout1-ec2-va.apache.org (mxout1-ec2-va.apache.org [3.227.148.255]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id 3360D18062C for ; Mon, 1 Mar 2021 07:05:02 +0100 (CET) Received: from mail.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mxout1-ec2-va.apache.org (ASF Mail Server at mxout1-ec2-va.apache.org) with SMTP id 6D3C241C4C for ; Mon, 1 Mar 2021 06:05:01 +0000 (UTC) Received: (qmail 90152 invoked by uid 500); 1 Mar 2021 06:05:01 -0000 Mailing-List: contact issues-help@beam.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@beam.apache.org Delivered-To: mailing list issues@beam.apache.org Received: (qmail 90127 invoked by uid 99); 1 Mar 2021 06:05:01 -0000 Received: from ec2-52-204-25-47.compute-1.amazonaws.com (HELO mailrelay1-ec2-va.apache.org) (52.204.25.47) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Mar 2021 06:05:01 +0000 Received: from jira2-he-de.apache.org (jira2-he-de.apache.org [168.119.33.54]) by mailrelay1-ec2-va.apache.org (ASF Mail Server at mailrelay1-ec2-va.apache.org) with ESMTPS id EC67241943 for ; Mon, 1 Mar 2021 06:05:00 +0000 (UTC) Received: from jira2-he-de.apache.org (localhost.localdomain [127.0.0.1]) by jira2-he-de.apache.org (ASF Mail Server at jira2-he-de.apache.org) with ESMTP id 193A0C801C4 for ; Mon, 1 Mar 2021 06:05:00 +0000 (UTC) Date: Mon, 1 Mar 2021 06:05:00 +0000 (UTC) From: "ASF GitHub Bot (Jira)" To: issues@beam.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Work logged] (BEAM-11277) WriteToBigQuery with batch file loads does not respect schema update options when there are multiple load jobs MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/BEAM-11277?focusedWorklogId=3D= 559202&page=3Dcom.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpa= nel#worklog-559202 ] ASF GitHub Bot logged work on BEAM-11277: ----------------------------------------- Author: ASF GitHub Bot Created on: 01/Mar/21 06:04 Start Date: 01/Mar/21 06:04 Worklog Time Spent: 10m=20 Work Description: codecov[bot] edited a comment on pull request #1411= 3: URL: https://github.com/apache/beam/pull/14113#issuecomment-787047099 # [Codecov](https://codecov.io/gh/apache/beam/pull/14113?src=3Dpr&el=3Dh= 1) Report > Merging [#14113](https://codecov.io/gh/apache/beam/pull/14113?src=3Dpr= &el=3Ddesc) (6c028ba) into [master](https://codecov.io/gh/apache/beam/commi= t/07e1e02125082d9ec804428f139eb849d79a8ec8?el=3Ddesc) (07e1e02) will **decr= ease** coverage by `0.00%`. > The diff coverage is `n/a`. =20 [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/1411= 3/graphs/tree.svg?width=3D650&height=3D150&src=3Dpr&token=3DqcbbAh8Fj1)](ht= tps://codecov.io/gh/apache/beam/pull/14113?src=3Dpr&el=3Dtree) =20 ```diff @@ Coverage Diff @@ ## master #14113 +/- ## =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D - Coverage 83.01% 83.01% -0.01% =20 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Files 469 469 =20 Lines 58355 58527 +172 =20 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + Hits 48443 48585 +142 =20 - Misses 9912 9942 +30 =20 ``` =20 =20 | [Impacted Files](https://codecov.io/gh/apache/beam/pull/14113?src=3Dpr= &el=3Dtree) | Coverage =CE=94 | | |---|---|---| | [...on/apache\_beam/runners/direct/watermark\_manager.py](https://code= cov.io/gh/apache/beam/pull/14113/diff?src=3Dpr&el=3Dtree#diff-YmVhbV9QcmVDb= 21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1= aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvd2F0ZXJtYXJ= rX21hbmFnZXIucHk=3D) | | | | [.../examples/snippets/transforms/elementwise/pardo.py](https://codeco= v.io/gh/apache/beam/pull/14113/diff?src=3Dpr&el=3Dtree#diff-YmVhbV9QcmVDb21= taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aW= xkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb= 3Jtcy9lbGVtZW50d2lzZS9wYXJkby5weQ=3D=3D) | | | | [...sdks/python/apache\_beam/runners/direct/\_\_init\_\_.py](https://c= odecov.io/gh/apache/beam/pull/14113/diff?src=3Dpr&el=3Dtree#diff-YmVhbV9Qcm= VDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L= 2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvX19pbml0= X18ucHk=3D) | | | | [...ild/srcs/sdks/python/apache\_beam/utils/sentinel.py](https://codec= ov.io/gh/apache/beam/pull/14113/diff?src=3Dpr&el=3Dtree#diff-YmVhbV9QcmVDb2= 1taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1a= WxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvc2VudGluZWwucHk=3D) | | = | | [...python/apache\_beam/examples/wordcount\_debugging.py](https://code= cov.io/gh/apache/beam/pull/14113/diff?src=3Dpr&el=3Dtree#diff-YmVhbV9QcmVDb= 21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1= aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvd29yZGNvdW50X2RlYnV= nZ2luZy5weQ=3D=3D) | | | | [...hon/apache\_beam/io/gcp/datastore/v1new/\_\_init\_\_.py](https://c= odecov.io/gh/apache/beam/pull/14113/diff?src=3Dpr&el=3Dtree#diff-YmVhbV9Qcm= VDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L= 2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2RhdGFzdG9yZS92MW5l= dy9fX2luaXRfXy5weQ=3D=3D) | | | | [.../python/apache\_beam/portability/api/metrics\_pb2.py](https://code= cov.io/gh/apache/beam/pull/14113/diff?src=3Dpr&el=3Dtree#diff-YmVhbV9QcmVDb= 21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1= aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3N= fcGIyLnB5) | | | | [...s/sdks/python/apache\_beam/transforms/ptransform.py](https://codec= ov.io/gh/apache/beam/pull/14113/diff?src=3Dpr&el=3Dtree#diff-YmVhbV9QcmVDb2= 1taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1a= WxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wdHJhbnNmb3JtLnB5= ) | | | | [...\_beam/testing/benchmarks/nexmark/queries/query4.py](https://codec= ov.io/gh/apache/beam/pull/14113/diff?src=3Dpr&el=3Dtree#diff-YmVhbV9QcmVDb2= 1taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1a= WxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1h= cmsvcXVlcmllcy9xdWVyeTQucHk=3D) | | | | [...y38/build/srcs/sdks/python/apache\_beam/pipeline.py](https://codec= ov.io/gh/apache/beam/pull/14113/diff?src=3Dpr&el=3Dtree#diff-YmVhbV9QcmVDb2= 1taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1a= WxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcGlwZWxpbmUucHk=3D) | | | | ... and [928 more](https://codecov.io/gh/apache/beam/pull/14113/diff?s= rc=3Dpr&el=3Dtree-more) | | =20 ------ =20 [Continue to review full report at Codecov](https://codecov.io/gh/apache= /beam/pull/14113?src=3Dpr&el=3Dcontinue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/c= odecov-delta) > `=CE=94 =3D absolute (impact)`, `=C3=B8 =3D not affected`, = `? =3D missing data` > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14113?src= =3Dpr&el=3Dfooter). Last update [07e1e02...6c028ba](https://codecov.io/gh/a= pache/beam/pull/14113?src=3Dpr&el=3Dlastupdated). Read the [comment docs](h= ttps://docs.codecov.io/docs/pull-request-comments). =20 ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: users@infra.apache.org Issue Time Tracking ------------------- Worklog Id: (was: 559202) Time Spent: 2h 20m (was: 2h 10m) > WriteToBigQuery with batch file loads does not respect schema update opti= ons when there are multiple load jobs > -------------------------------------------------------------------------= ------------------------------------- > > Key: BEAM-11277 > URL: https://issues.apache.org/jira/browse/BEAM-11277 > Project: Beam > Issue Type: Bug > Components: io-py-gcp, runner-dataflow > Affects Versions: 2.21.0, 2.24.0, 2.25.0 > Reporter: Chun Yang > Assignee: Chun Yang > Priority: P2 > Attachments: repro.py > > Time Spent: 2h 20m > Remaining Estimate: 0h > > When multiple load jobs are needed to write data to a destination table, = e.g., when the data is spread over more than [10,000|https://cloud.google.c= om/bigquery/quotas#load_jobs] URIs, WriteToBigQuery in FILE_LOADS mode will= write data into temporary tables and then copy the temporary tables into t= he destination table. > When WriteToBigQuery is used with {{write_disposition=3DBigQueryDispositi= on.WRITE_APPEND}} and {{additional_bq_parameters=3D\{"schemaUpdateOptions":= ["ALLOW_FIELD_ADDITION"]\}}}, the schema update options are not respected = by the jobs that copy data from temporary tables into the destination table= . The effect is that for small jobs (<10K source URIs), schema field additi= on is allowed, however, if the job is scaled to >10K source URIs, then sche= ma field addition will fail with an error such as: > {code:none}Provided Schema does not match Table project:dataset.table. Ca= nnot add fields (field: field_name){code} > I've been able to reproduce this issue with Python 3.7 and DataflowRunner= on Beam 2.21.0 and Beam 2.25.0. I could not reproduce the issue with Direc= tRunner. A minimal reproducible example is attached. -- This message was sent by Atlassian Jira (v8.3.4#803005)